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Live  Fire  Testing  and  Evaluation— a  critical  component  in  the 
overall  success  of  the  F-35  fighter  jet  program. 
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Several  key  Live  Fire  Testing  and  Evaluation  (LFT&E) 
lessons  were  learned  from  the  prime  contractor 
perspective  on  the  F-35/Joint  Strike  Fighter  (JSF) 
program.  The  F-35  fighter  jet  effort  includes  three 
variants  that  increase  overall  program  complexity 
and  risk.  The  LFT&E  component  of  this  development  effort  is 
critical  to  the  overall  success  of  the  F-35  program. 


Evaluation  Lessors 
From  Live  Fire  Testing 

The  F-35  Lightning  II-JoinFStrike 
Fighter  Program 


Steve  Mills  ■  Mark  Stewart 


LFT&E  is  a  critical  element  of  the  system  engineering  and  test  and  evaluation  processes  for  Department  of  Defense 
(DoD)  systems.  The  current  ACQuipedia  article  on  LFT&E  provides  this  straightforward  explanation  of  LFT&E  as 
part  of  the  DoD  acquisition  process: 


Mills  is  a  professor  of  Program  Management  and  Cybersecurity  at  the  Defense  Acquisition  University's  South  Region  in  Huntsville ,  Alabama. 
Stewart  is  a  Lockheed  Martin  Fellow  specializing  in  Aircraft  Survivability/Vulnerability  and  Technical  Lead  of  the  F-35  Vulnerability  Analysis 
and  Live  Fire  Test  Team. 
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A  test  p 

lethality  aspects  of  a  conventional  weapon  or  conven¬ 
tional  weapon  system.  LFT&E  is  a  statutory  requirement 
(Title  10  U.S.C.  [U.S.  Code]  §  2366)  for  covered  sys¬ 
tems,  major  munitions  programs,  missile  programs,  or 
product  improvements  to  a  covered  system,  major  mu¬ 
nitions  programs,  or  missile  programs  before  they  can 
proceed  Beyond  Low  Rate  Initial  Production  (BLRIP).  By 
law,  a  covered  system  is  any  vehicle,  weapon  platform, 
or  conventional  weapon  system  that  includes  features 
designed  to  provide  some  degree  of  protection  to  users 
in  combat  and  that  is  an  Acquisition  Category  (ACAT)  I 
or  ACAT  II  program.  (Note:  The  term  "covered  system" 
can  also  be  taken  to  mean  any  system  or  program  that  is 
covered  by  Title  10  U.S.C.  §  2366,  including  major  muni¬ 
tions  and  missile  programs.) 


LFT&E  focuses  on  evaluating  the  survivability  and  le¬ 
thality  of  a  system.  With  regard  to  the  F-35  program, 


As  the  DoD  moves  closer  to  a  full-rate  production  deci¬ 
sion  for  F-35,  the  lessons  learned  from  the  LFT&E  efforts 
of  the  F-35  industry  team  led  by  the  prime  contractor, 
Lockheed  Martin  Corporation,  can  provide  other  acqui¬ 
sition  organizations  with  valuable  insight  into  how  best 
to  conduct  LFT&E  on  their  respective  programs. 


LFT&E  Lessons  Learned  From 
an  Industry  Perspective 

The  F-35  LFT&E  program  was  one  of  the  most  compre¬ 
hensive  in  fixed-wing  procurements.  The  F-35  program 
consisted  of  61  test  series,  with  more  than  1,500  events 
against  ballistic  threats.  The  F-35  LFT&E  lessons  learned 


these  two  attributes  are  paramount  to  the  success  of  this 
system  operating  in  its  intended  environment.  Although 
they  are  similar,  each  F-35  variant  has  its  own  unique 
survivability  and  lethality  requirements  as  well,  making 
this  an  even  bigger  challenge  to  getting  it  right. 
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Figure  1.  Formal  Lines  of  LFT&E  Figure  2.  Informal  Lines  of  LFT&E  Commu- 
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from  the  prime  contractor  perspective  may  be  grouped  into 
two  general  categories:  Government-Industry  Teamwork  and 
Limiting  Scope. 

Government-Industry  Teamwork 

Lines  of  Communication.  LFT&E  requires  the  efforts  of  at 
least  four  primary  entities:  The  Program  Office,  Director; 
Operational  Test  and  Evaluation  (DOT&E)  Representatives; 
the  primary  weapon  system  contractors;  and  the  government 
test  facility  organizations.  Contractual  relationships  provide  a 
formal  flow  between  the  four  primary  entities  involved  in  Live 
Fire  Testing  (LFT)  (Figure  1),  but  the  formal  lines  of  commu¬ 
nication  lack  the  ability  to  build  a  team  capable  of  effectively 
and  efficiently  executing  the  LFT  Program. 

In  a  previous  LFT&E  program,  the  prevailing  wisdom  was  to 
keep  control  of  the  program  by  only  allowing  the  formal  lines 
of  communication  between  entities;  in  particular,  the  intention 
was  to  limit  communication  to  DOT&E  representatives.  Much 
of  the  success  of  the  F-35  LFT&E  program  can  be  attributed  to 
the  open  communication  and  informal  information  flows  that 
were  created  and  maintained  throughout  (Figure  2). 

Diverse  Organizations.  Each  of  these  organizations  have  dif¬ 
fering  goals  and  constraints  that  sometimes  make  teamwork 
difficult.  Industry  partners  want  to  limit  company  costs  and 


risk  while  meeting  contractual  obligations.  The  Program  Of¬ 
fice  wants  to  limit  program  impacts  while  delivering  value  to 
the  warfighter.  DOT&E's  objective  is  to  thoroughly  test.  Their 
success  is  sometimes  dependent  on  "findings":  discovering 
shortfalls  or  unexpected  results.  Ultimately,  these  disparate 
organizations  must  come  together  to  produce  a  test  program 
that  meets  the  objectives  while  living  within  the  constraints.  It 
is  important  that  each  organizational  member  is  at  least  made 
aware  of  the  varying  goals  of  the  other  members. 

Roles  and  Responsibilities.  It  is  imperative  that  roles  and 
responsibilities  are  established  early  in  the  program.  On  F-35, 
we  determined  that  the  prime  contractor  should  have  the 
responsibility  to  create  all  test  plans  and  reports.  Test  ar¬ 
ticle  construction  is  an  activity  that  should  be  shared  by  the 
contractor  and  test  facilities.  The  government  test  facilities 
have  tremendous  abilities  to  quickly  design  and  construct 
test  articles,  particularly  if  these  articles  are  sub-assemblies, 
and  not  required  to  be  production  representative.  The  con¬ 
tractors  must  design  and  construct  the  more  complicated 
articles,  but  can  be  less  efficient  in  building  the  simple  ones. 
The  test  facilities  must  be  the  final  technical  approval  for  the 
test  plans  as  they  are  the  ones  that  must  ultimately  execute 
the  test.  DOT&E  representatives  must  provide  timely  reviews 
and  constructive  comments  on  each  test  plan,  along  with 
formal  signoff  on  the  plans  utilizing  full-scale  and  Full-Up, 
System-Level  articles. 

Establishing  Trust.  Trust  between  team  members  will  be  es¬ 
tablished  only  through  time,  with  open  communication,  hon¬ 
est  discussions,  and  mutual  respect.  On  F-35,  we  established 
weekly  telephone  calls  with  all  organizations  to  foster  trust. 
We  also  established  a  collaborative  workspace  on  the  F-35 
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Data  Library  in  order  to  share  technical  information  in  the  form 
of  test  plans,  analyses  and  test  reports.  These  two  forums 
kept  the  informal  lines  of  communication  open  throughout 
the  program. 

Limiting  the  Scope 

Scope  Creep.  One  of  the  most  difficult  problems  encountered 
with  F-35  LFT&E  was  the  need  to  prevent  scope  creep.  The 
cost  of  each  test  series  is  highly  dependent  on  the  objectives, 
test  matrix,  and  complexity  of  the  test  article.  Open  com¬ 
munication  helps  in  that  each  team's  organization  is  able  to 
express  their  objectives,  concerns  and  constraints.  The  objec- 


F-35  after  loss  of  a  complete  or  partial  control  surface.  Man- 
in-the-loop  simulation  was  used  to  verify  the  loss  of  multiple 
flight  control  and  electrical  power  components  that  were  in 
close  proximity.  The  F-35's  Fuel  System  Simulator  was  used 
to  provide  data  on  fuel  loss  and  fuel  tank  explosion  preven¬ 
tion.  These  tests  were  much  less  expensive  than  some  of  the 
more  complex  ballistic  test  articles,  and  provided  a  wealth  of 
information  to  address  many  LFT&E  issues. 

Modeling  and  Simulation  (M&S).  M&S,  once  shunned  by 
the  Live  Fire  Test  Office  within  OT&E,  is  now  being  used  in 
a  significant  way.  In  the  F-35  program,  a  symbiotic  relation- 
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tives  of  the  test,  if  defined  in  detail,  will  allow  the  team  to  limit 
the  complexity  of  the  test  article.  Spares  to  replace  damaged 
components  must  also  be  taken  into  account.  The  order  of 
the  events  on  each  test  article  requires  much  coordination  but 
will  yield  the  most  data  without  requiring  needless  replace¬ 
ments  and  repairs.  There  was  a  transition  from  simple  Test 
Data  Sheets  on  previous  programs  to  full-up  Test  Plans  which 
were  used  on  the  F-35;  these  comprehensive  documents  went 
a  long  way  to  limiting  objectives  and  setting  expectations  for 
each  test  series. 

Objectives.  LFT&E  issues  and  sub-issues  are  provided  to  the 
contractors  via  the  Test  and  Evaluation  Master  Plan  (TEMP), 
but  the  issues  provided  are  very  general  in  nature,  leaving 
much  room  for  interpretation.  Therefore,  each  F-35  test  se¬ 
ries  was  designed  to  specifically  address  a  particular  set  of 
objectives  within  the  list  of  LFT&E  sub-issues. 

Threats.  Similarly,  the  potential  threat  list  is  also  daunting.  It 
was  important  early  on  to  define  threat  types  and  to  limit  the 
scope  by  addressing  only  those  potential  threats  (with  some 
exceptions).  An  underlying  purpose  of  the  F-35  LFT&E  plan 
was  that  the  tests  were  going  to  meet  the  objectives  in  the 
TEMP  as  well  as  provide  insights  to  the  F-35  design  team.  The 
tests  would  also  provide  missing/inadequate  data  to  improve 
the  F-35  vulnerability  analysis. 

Controlled  Damage  Tests.  The  F-35  LFT&E  team  utilized  more 
than  just  ballistic  tests  to  address  LFT&E  issues.  Wind  Tunnel 
tests  were  conducted  to  determine  the  controllability  of  the 


ship  was  established  early  in  the  program  between  M&S  and 
LFT&E.  Early  in  the  program,  the  contractor  team  conducted 
a  unique  vulnerability  uncertainty  analysis  that  gave  insights 
into  which  damage  mechanisms  had  the  largest  potential  to 
affect  the  F-35's  vulnerability  assessment  results.  These  un¬ 
certainties  were  folded  into  the  LFT&E  test  plans,  and  provided 
a  context  for  discussions  concerning  the  relative  importance 
of  individual  test  events.  For  example,  testing  to  determine  the 
vulnerability  of  the  F-35's  flight  control  computers  was  deter¬ 
mined  to  be  relatively  unimportant  due  to  system  redundancy, 
which  rendered  the  potential  for  loss-of-aircraft  to  be  relatively 
small.  All  test  events  were  preceded  by  test  predictions,  most 
of  which  were  conducted  via  M&S.  Tests  verified  the  ability  of 
the  M&S  in  some  cases,  while  providing  critical  data  to  allow 
improvements  to  be  made. 

Conclusion 

DoD  acquisition  program  success  hinges  on  the  partnership 
between  both  government  and  industry  in  the  execution  of  a 
robust  systems  engineering  process  to  deliver  effective  so¬ 
lutions  to  the  warfighter.  LFT&E  is  a  key  component  of  this 
systems  engineering  effort.  This  article  offers  valuable  les¬ 
sons  learned  from  the  industry  partner  perspective  on  how 
to  effectively  execute  LFT&E  on  a  very  complex  acquisition 
program— the  F-35.  In  the  end,  the  success  of  our  efforts,  both 
government  and  industry,  will  be  based  on  our  strong  part¬ 
nership,  effective  communication  and  teamwork  to  meet  the 
needs  of  the  warfighter. 
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The  authors  can  be  contacted  at  steve.mills@dau.mil  and 
mark.w.stewart@lmco.com. 
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AUTHORS’ NOTE 

The  following  is  a  fictionalized  representation  of  real  cybersecurity  issues  encountered  in  the  Department  of  Defense 
(DoD)  and  is  a  sequel  to  'The  Quest  for  Defense  Cybersecurity"  article  published  in  the  November-December  2017 
issue  of  Defense  AT&L  (https://www.dau.mil/library/defense-atl/blog/The-Quest-for-Defense-Cybersecurity).  In 
that  earlier  article,  the  authors  examined  a  process  to  identify  vulnerabilities  and  develop  requirements  needed 
to  begin  to  execute  on  the  DoD's  six-phase  cybersecurity  Test  and  Evaluation  process.  In  this  article,  the  authors 
expand  their  argument  to  address  instilling  a  "culture  of  cyber  awareness  [that]  must  permeate  into  all  facets  of 
weapons  systems  acquisition,  training,  maintenance,  and  operations." 


Reuters  news  service,  January  2020.  "The  USS  Jimmy  Doolittle ,  the  U.S.  Navy's  newest 
and  largest  nuclear  powered  aircraft  carrier,  was  recently  subjected  to  an  intense  'cyber 
attack'  from  a  non-nation-state  actor.  However,  due  to  efforts  to  understand  cyber  vulner¬ 
abilities  and  anticipate  the  effects  of  successful  cyber  attacks  early  in  the  USS  Doolittle's 
development,  this  attack  was  largely  mitigated  and  the  combat  elements  of  the  Doolittle 
were  still  able  to  carry  out  their  missions  successfully.'' 
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This  fictional  new  aircraft  carrier,  the  Doolittle ,  is  1,156 
feet  long,  has  a  beam  of  150  feet  at  the  waterline  and 
displaces  just  over  101,000  tons.  The  Doolittle's  mis¬ 
sion  is  to  project  national  power  and  destroy  or  neu¬ 
tralize  enemy  targets  ashore  and  at  sea.  Specific  tasks 
include  Air,  Surface,  and  Antisubmarine  Warfare,  Com¬ 
mand,  Control,  and  Communications  (C3),  Command 
and  Control  Warfare  (C2W),  Intelligence,  Mine  War¬ 
fare  and  Strike  Warfare.  This  is  in  addition  to  the  ship 
performing  Fleet  Support  Operations,  Logistics,  Non- 


Combat  Operations,  and  Naval  Special  Warfare.  In 
addition  to  the  systems  required  to  perform  the  above 
missions  and  tasks,  it  requires  a  secure  Command, 
Control,  Communications,  Computers,  and  Intelligence 
(C4I)  system,  enclaves  for  Unclassified,  Coalition,  Se¬ 
cret  and  Sensitive  Compartmented  Information  (SCI) 
environments.  It  has  a  common  computer  domain  for 
conducting  command,  control,  intelligence,  business, 
maintenance,  supply,  and  air  wing  operations.  In  ad¬ 
dition,  the  Doolittle  must  communicate  with  myriad 


Thompson,  a  retired  U.S.  Air  Force  ( USAF )  colonel,  is  director  of  Cyber  and  Air  Force  programs  at  Electronic  Warfare  Associates,  Inc.  (EWA), 
in  Herndon,  Virginia.  He  is  a  graduate  of  the  USAF  Test  Pilot  School  and  holds  a  Master  of  Science  in  Systems  Engineering  from  the  Air  Force 
Institute  of  Technology.  Lilienthal  is  the  director  of  Cyber  and  Navy  Programs  at  EWA.  He  has  a  doctorate  in  Experimental  Psychology  from 
the  University  of  Notre  Dame.  He  served  for  more  than  30  years  as  a  Navy  aerospace  experimental  psychologist  and  worked  in  program 
management,  test  and  evaluation  (T&E),  and  training.  He  is  a  retired  U.S.  Navy  captain.  Brown,  a  retired  USAF  colonel,  is  EWA's  director 
for  Cyber  Programs.  A  graduate  of  the  USAF  Fighter  Weapons  School,  he  retired  as  a  Command  fighter  pilot  after  30  years  of  service  in  both 
operations  and  T&E. 
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support  systems.  Many  of  these  support  and  subsystems  are 
legacy  to  the  Navy  and  were  designed  without  consideration 
to  cyberattack.  And  all  these  systems  and  subsystems  are 
subject  to  routine  software  upgrades. 

*  *  * 

U  Greetings,  shipmates!  I'm  LT  Bart  Savagewood, 
USN,  and  I  fly  F/A-18  Super  Hornets  on  the  USS 
Jimmy  Doolittle.  I've  been  asked  to  write  a  few  words 
about  cyber  and  what  it  is  I  believe  is  important  in  the  cyber 
world.  As  a  Nugget,  or  new  naval  aviator  on  my  first  cruise, 
I  didn't  know  much  about  cyber  stuff  or  even  care.  As  I  pro¬ 
gressed  to  lieutenant  junior  grade,  I  knew  that  some  folks  in 
the  Navy,  the  cyber  geeks,  I  mean  cyber  warriors,  were  wor¬ 
rying  about  cyber.  And  not  too  long  ago,  after  I  progressed  in 
rank  to  lieutenant,  I  heard  that  the  bosses  had  to  worry  about 
something  called  Section  1647  of  the  National  Defense  Autho¬ 
rization  Act  for  Fiscal  Year  2016.  But  not  me!  I'm  an  operator! 
I  mean,  I  fly  Hornets  off  carriers  and  kill  bad  guys,  so  why  did 
I  need  to  know  about  cyber? 

"Besides,  I  am  all  about  following  the  cyber  rules,  and  com¬ 
pliance  is  my  middle  name.  I  know  that  I  am  not  supposed  to 
use  thumb  drives  in  my  Navy-issued  laptops,  and  I  hardly  ever 
do.  I  know  if  I  am  caught,  I  will  get  locked  out  of  the  carrier's 
system,  which  is  nothing  compared  to  what  the  Skipper  will  do 
to  me.  Of  course,  I  know  that  I  should  immediately  delete  any 
unauthorized  e-mails,  'cause  if  I  open  an  unauthorized  e-mail 
on  my  government  computer  and  it  contains  a  virus,  I  will  be 
condemned  to  a  penalty  box  for  up  to  2  days  of  what  they  call 
Information  Assurance  (IA)  'refresher  training.'  What  a  pain! 
I  know  that  I  am  not  supposed  to  use  my  personal  devices 
like  iPods  on  any  Navy-issued  computer.  And,  like  with  thumb 
drives,  I  hardly  ever  do  so.  I  know  I  am  supposed  to  follow 
certain  rules  on  surfing  the  Internet  and  then  downloading 
material  onto  Navy  computers.  But  when  the  Executive  Officer 
wants  to  have  a  video  for  the  Ready  Room  by  this  evening- 
well,  sometimes  you  'gotta  do  what  you  gotta  do.' 

"But  why  should  all  the  cyber  heat  come  down  on  the  opera¬ 
tors  like  me?  We  use  laptops  that  are  15  years  old  and  they  run 
on  Windows  XP  software  and  don't  even  have  DVD  capability. 
Our  Fitness  Reports,  Annual  Officer  Evaluation  Reports  are 
created  in  NAVFIT98A— yes,  a  computer  program  from  a  gen¬ 
eration  ago  that  runs  on  Windows  VISTA  and  XP— whatever 
they  are!  Our  flight  logging  program,  SHARP,  is  only  32  bit, 
which  is  in  the  Stone  Age  compared  to  the  Air  Force.  The  In¬ 
ternet  speed  onboard  ship  is  pretty  bad  unless  you  have  com¬ 
manding  officer  or  department  head  privileges.  So  it  is  nearly 
impossible  for  a  junior  officer  (J.O.)  as  the  squadron  duty  of¬ 
ficer  to  access  weather/Notices  to  Airmen  for  flight  briefings. 
Of  course,  to  even  access  a  Navy  Information  Technology  (IT) 
system,  you  must  do  annual  Information  Assurance  (IA)  train¬ 
ing  that  is  like  a  terrible  videogame  that  hasn't  changed  in  5 
years.  Don't  they  realize  that  everyone  just  speed  clicks  the 
training  and  retains  nothing  from  it?  And  why  do  they  even 
still  use  the  term  IA?  Wasn't  it  supposed  to  go  away  a  while 


back  when  they  published  Department  of  Defense  Instruction, 
Number  8500.01  (March  14,  2014),  which  adopted  the  term 
'cybersecurity'  and  directed  that  'Information  Assurance  (IA) 
Implementation'  be  canceled? 

"Deploying  aboard  the  ship  always  is  an  IT  nightmare.  IT  is 
supposed  to  migrate  your  shore  Outlook,  share  drive,  and  e- 
mail,  but  I've  never  seen  it  work  very  well.  You  basically  have 
to  burn  anything  important  to  a  CD  and  take  it  with  you  so  you 
don't  lose  all  the  projects  you've  been  working  on.  And  why 
can't  cyber  and  IT  be  friendlier  to  operators  like  me?  I  think 
they  mandated  the  15-character  passwords  that  are  impos¬ 
sible  to  remember,  and  have  to  be  changed  every  60  days, 
just  to  make  life  difficult  for  us.  Now  the  IT  guys  are  preach¬ 
ing  to  me  about  something  they  call  'cyber  hygiene.'  I'm  not 
really  sure  what  that  even  means  other  than  adding  yet  more 
roadblocks  and  inconveniences  to  my  computer. 

"So  what  is  it  that  I  want  from  the  cyber  community?  To  be 
honest,  as  a  J.O.,  I  would  have  said  all's  I  wanted  out  of  the 
cyber  geeks  was  for  them  to  get  out  of  my  way  and  to  quit 
making  my  job  harder.  Now,  I  want  nothing  less  than  a  culture 
change  in  the  way  the  Navy  approaches  developing  systems 
and  adopting  operations  to  succeed  in  a  cyber  world. 

"What  made  me  change  my  perspective?  After  two  deploy¬ 
ments  on  another  aircraft  carrier  and  a  stint  as  an  instructor 
at  the  RAG  (Replacement  Air  Group),  I  was  stationed  onboard 
the  USS  Jimmy  Doolittle,  and  things  changed  in  my  cyber  world. 
As  I  came  onboard  the  Doolittle,  as  we  affectionately  call  her,  I 
started  hearing  a  lot  about  cyber.  I  was  told  that  the  ship's  de¬ 
signers  and  program  managers  knew  that  this  complex  family- 
of-systems  could  have  been  a  cybersecurity  nightmare.  They 
knew  that  compliance  with  the  Navy's  CYBERSAFE  program 
would  guide  them  to  'provide  maximum  reasonable  assurance 
of  survivability  and  resiliency  of  mission  critical  information 
technology,  in  a  contested  cyber  environment  in  order  to 
maintain  mission  capabilities.'  But  more  than  that,  they  knew 
they  needed  to  instill  a  new  culture  of  'cyber  resilience,'  or 
the  ability  to  successfully  execute  operations  in  a  contested 
cyber  environment  into  all  facets  of  ship  design,  development, 
testing  and  operations. 

"So,  very  early  in  the  Doolittle's  concept  development  and  de¬ 
sign  phase,  the  ship's  planners  brought  together  operators, 
maintainers,  systems  engineers,  testers,  and  cyber  experts 
to  not  simply  take  the  approach  of  compliance  with  current 
checklist  directives  and  policies  but  to  approach  the  design, 
operation,  and  maintenance  aboard  the  USS  Doolittle  from  a 
mission  viewpoint.  To  do  that,  they  began  a  disciplined  pro¬ 
cess  they  called  a  Cyber  Operational  Vulnerability  Assessment 
(COVA).  The  Doolittle  COVA  is  a  rigorous  process  leveraging 
war-gaming  principals  that  focus  on  developing  an  under¬ 
standing  of: 

■  How  personnel  actually  use  and  maintain  a  system  to 
carry  out  a  specific  mission 
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To  be  honest,  as  a  J.O.,  I  would  have  said  airs  I 
wanted  out  of  the  cyber  geeks  was  for  them  to  get  out  of  my  way  and 
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quit  making  my  job  harder.  NowJ  want  nothing  less  than  a 
culture  change  in  the  way  the  Navy  approaches  developing  systemF 


and  adopting  operations  to  succeed  in  a  cyber  world. 
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■  How  successful  cyber  attacks  degrade  or  prevent  opera¬ 
tional  mission  success 

■  And  how  potential  actions  or  workarounds  might  prevent 
or  minimize  cyber  effects. 

"Leveraging  the  COVA  results,  the  USS  Doolittle  managers 
ensured  the  engineers  and  cybersecurity  personnel  worked 
with  those  with  fleet  operational  experience  so  both  would 
have  a  clear  understanding  of  the  technological  capabilities 
of  the  new  system(s). 

"The  managers  demanded  all  shipboard  disciplines  work  as 
one  team  to  understand  potential  cyber  effects  and  mission 
consequences.  Because  they  routinely  participated  in  onboard 
COVA  events,  the  Doolittle's  cyber  warriors  now  understood 
the  mission,  the  operational  environment  and  how  it  might  be 
affected  by  their  controls  and  protections.  The  operators,  like 
me,  but  also  including  maintainers,  supply,  ship  drivers,  snipes, 
etc.,  now  understand  the  potential  for  cyber  effects— mean¬ 
ing  they  understand  the  controls  and  protections  needed  for 
their  own  mission  success.  Together,  the  cyber  and  operations 
communities  were  able  to  effectively  communicate  to  program 
managment  the  risks,  costs,  limitations,  and  alternatives  of 
protections  and  controls. 


shipboard  operations,  including  temporarily  assigned  air  wings 
and  support  assets.  It  is  from  this  perspective  as  a  tactical 
operator  onboard  the  USS  Jimmy  Doolittle  that  I  say  I  want  a 
cyber  culture  change  in  the  Navy.  The  culture  change  I  want 
is  one  that  will  embed  cyber  considerations  into  all  aspects  of 
operations  with  a  focus  on  mission  impact.  I  want  the  cyber 
warriors  to  understand  what  I  do.  They  need  to  understand 
how  cyber  protections  affect  operators.  And  the  reverse  also 
is  true.  Operators,  maintainers,  logistics,  and  all  support  folks 
need  to  understand  cyber  effects  and  how  they  can  influence 
offensive  actions  as  well  as  defensive  operational  impacts. 

"There  are  many  offensive  and  defensive  cyber  capabilities 
available  for  operations  onboard  the  USS  Doolittle.  But  the  very 
nature  of  many  of  these  capabilities  means  that  they  will  con¬ 
tinue  to  be  held  at  the  upper  echelons  of  Naval  and  National 
Command.  Are  there  specific  capabilities  that  I  want  at  the 
tactical  level?  Of  course!  But  until  there  is  a  culture  change 
within  the  Navy  and  other  Services,  cyber  will  continue  to  be  a 
friction  point  within  our  own  operations.  A  culture  change  like 
the  one  I  want  will  provide  a  comprehensive  cyber 
focus  on  mission  accomplishment  by  aimingto  de¬ 
tect  and  minimize  mission  impact  of  cyberattacks. 


"Captializing  on  this  relationship,  potential  workarounds  and 
engineering  options  were  continously  developed  and  evalu¬ 
ated  throughout  the  acqusition  and  development  process.  The 
ship's  designers  and  operators  assumed  they  were  going  to  be 
in  a  cyber-contested  environment;  that  cyber  hackers  would 
find  new  and  innovative  ways  to  penetrate  vulnerabilities  and 
weaknesses;  that  all  software  and  firmware  were  flawed,  and 
personnel  who  operated  the  USS  Doolittle  would  make  mis¬ 
takes  that  would  enable  a  cyber  attack.  They  looked  at  designs 
and  design  trade-offs  early  with  that  in  mind.  As  system  design 
progressed,  they  continued  the  iterative  COVA  process  to  in¬ 
clude  the  more  mature  versions  of  systems  and  added  addi¬ 
tional  systems  to  the  process  to  insure  operational  relevance. 


As  a  sidebar  related  to  changing  the  Navy's  cyber  culture,  I  be¬ 
lieve  that  Electronic  Warfare  needs  to  be  considered  in  tandem 
with  cyber  warfare.  The  use  of  the  Electromagnetic  Spectrum 
(EMS)  can  be  affected  or  disrupted  by  cyber  or  electronic  war¬ 
fare  domains.  The  EMS  is  critical  for  communications,  com¬ 
mand  and  control,  blue  force  tracking,  precision  attack,  and 
more  warfighting  capabilities.  Potential  adversaries  learned 
from  Desert  Storm  and  subsequent  engagements  how  the  U.S. 
military  uses  and  depends  on  EMS.  Today's  adversaries  know 
and  understand  the  EMS  and  will  contest  U.S.  military  access 
to  it.  The  Navy  and  other  Services  cannot  deal  with  each  war¬ 
fare  domain  separately;  they  must  be  viewed  as  complements 
of  each  other. 


"The  COVA  process  initiated  by  the  Doolittle  Program  Of¬ 
fice  was  intended  to  be  used  throughout  the  life  cycle  of 
the  Doolittle  program— from  concept  development  through 
operational  deployment  and  sustainment.  The  rigorous  and 
continued  use  of  the  COVA  process  incorporated  cyber  aware¬ 
ness  into  the  ship's  culture,  an  awareness  that  permeated  all 


Conclusion 

While  LT  Savagewood  and  the  USS  Doolittle  are  fictitious  tools 
for  this  essay,  the  solutions  discussed  to  implement  a  success¬ 
ful  cyber  culture  change  are  not.  The  COVA  described  in  this 
article  was  developed  on  the  foundation  of  a  cyber  tabletop 
process  the  U.S.  Naval  Air  Systems  Command  (NAVAIR)  has 
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adopted  as  a  standard  work  "best  practice"  for  determining 
cyber  vulnerabilities  and  requirements.  The  process  was  rec¬ 
ognized  by  NAVAIR  as  an  important  tool  in  operational  threat 
risk  assessment  as  well  as  a  catalyst  for  intellectual  change. 
A  senior  NAVAIR  director  offered  the  following  assessment 
following  a  recent  cyber  tabletop  excercise: 

The  event  was  a  "game  changer,"  in  that  it  not  only  helped 
identify  vulnerabilities,  but  it  tied  them  to  mission  risk  and  also 
helped  with  the  culture  change  necessary  to  get  our  entire  work¬ 
force  behind  this  important  topic.  Getting  our  engineers,  fleet, 
and  program  offices  to  understand  exactly  what  a  potential 
adversary  could  do  to  a  ship's  ability  to  safely  and  efficiently 
launch  and  recover  aircraft  was  worth  it  alone.  We  will  be  using 
the  results  from  this  event  to  drive  POM  [Program  Objective 
Memorandum]  requests,  recommend  technical  fixes,  plan  fur¬ 
ther  analysis/testing,  as  well  as  change  some  of  our  internal 
processes."  —by  permission,  June  12, 2017,  Kathleen  Donnelly, 
Senior  Executive  Service,  NAVAIR 4.8,  Director,  Support  Equip¬ 
ment  and  Aircraft  Launch  and  Recovery  Equipment. 


Is  this  culture  change  advocated  by  LT  Savagewood  unique 
to  naval  aviation,  the  U.S.  carrier  fleet  or  even  the  Navy?  Of 
course  not!  Substitute  LT  Savagewood  for  any  military  operator 
and  substitute  the  USS  Jimmy  Doolittle  for  any  DoD  acquisition 
program  and  the  analogy  fits.  The  key  to  achieving  a  cyber 
culture  change  within  various  Service  and  DoD  programs  is 
implementing  a  process  for  embedding  cybersecurity  across 
the  life  cycle  of  acquisition  design,  development,  testing,  and 
operational  employment.  A  culture  of  cyber  awareness  must 
permeate  all  facets  of  weapons  systems  acquisition,  training, 
maintenance,  and  operations.  A  process  similar  to  the  Doo¬ 
little's  COVA  must  be  iterative,  expeditious  and  readily  un¬ 
derstandable  to  the  operators  and  cyber  experts.  It  should  be 
implemented  early  and  continuously  across  the  acquisition 
and  operational  life  cycle  to  ensure  continued  success  in  a 
cyber  environment.  It's  past  time  to  get  started! 

The  authors  can  be  contacted  at  SThompson@ewa.com, 
MLilienthal@ewa.com  and  DBrown@ewa.com. 


Interested  in  passing  on  lessons  learned  from  your 
project  or  program? 

Willing  to  share  your  expertise  with  the 
acquisition  community? 


What’s  in  It  for  You? 

First  off,  seeing  your  name  in  print  is  quite  a  kick.  But  more 
than  that,  publishing  in  Defense  AT&L  can  help  advance  your 
career.  One  of  our  authors  has  even  been  offered  jobs  on  the 
basis  of  articles  written  for  the  magazine. 


Want  to  help  change  the  way  DoD  does  business? 

Write  an  article  (1,500  to  2,500  words)  and  Defense  AT&L 
will  consider  it  for  publication.  Our  readers  are  interested  in 
real-life,  hands-on  experiences  that  will  help  them  expand  their 
knowledge  and  do  their  jobs  better. 


Now  we  can’t  promise  you  a  new  job,  but  many  of  our  authors: 

•  Earn  continuous  learning  points 

•  Gain  recognition  as  subject-matter  experts 

•  Are  invited  to  speak  at  conferences  or  symposia 

•  Get  promoted  or  rewarded 


For  more  information  and  advice  on  how  to  submit  your  manuscript,  check  the  writer’s  guidelines  at 
https://www.dau.miI/library/defense-atl/p/Writers-Guidelines  or  contact  the  managing  editor  at  datl@dau.mil. 
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Department  of  Defense  (DoD)  acquisition  teams  currently  struggle  to  obtain 
clear  and  consistent  market  research  documentation  on  a  regular  basis. 
They  also  struggle  with  the  question  of  whether  market  research  should 
be  completed.  And  many  firms  or  agencies  sometimes  become  confused 
somewhere  between  the  market  research  and  the  solicitation  phases.  In 
2015,  the  Government  Accountability  Office  (GAO)  completed  a  review  of  28  DoD 
contracts,  at  least  50  percent  of  which  had  inadequate,  inconsistent  and  unclear 
market  research  documentation. 

In  2017,  the  International  Journal  of  Market  Research  noted  that  no  one  possesses  a  "crystal  ball"  for  obtaining  infor¬ 
mation  on  a  specific  market,  especially  given  the  regular  disruptions  or  changes  in  various  market  sectors  over  the 
last  30  years.  Does  your  acquisition  team  have  a  crystal  ball  for  obtaining  information  on  a  specific  business  market 
sector?  Are  your  team's  research  methodology  and  documentation  adequate  to  get  clear  and  concise  results? 


Cable  is  a  freelance  author ;  teacher  and  acquisition  professional.  He  holds  a  doctorate  of  Business  Administration  and  is  a  Certified  National 
Contract  Manager  (National  Contract  Management  Association). 
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For  an  interpreter  of  public  contract  law  and  regulations,  many 
more  catechisms  will  come  to  mind:  Do  acquisition  teams 
place  any  value  or  importance  in  completing  market  research? 
What  are  the  required  elements  in  market  research?  What  is 
the  proper  time  to  complete  market  research?  Should  market 
research  be  undertaken  at  all? 

Background 

The  governance  statute  of  market  research  comes  from 
Competition  in  Contracting  Act  of  1984  (CICA).  Many 
sources  state  that  the  empirical  requirement  of  market  re¬ 
search  comes  from  the  Federal  Acquisition  Streamline  Act 
of  1994,  which  only  clarified  the  differences  between  the 
commercial  and  noncommercial  items.  CICA  requires  re¬ 
search  of  a  market  for  competition  and  possible  sources  for  a 
requirement.  The  governance  regulation  for  market  research 
is  found  in  Chapter  10  of  the  Federal  Acquisition  Regulation 
(FAR).  According  to  FAR  Chapter  10,  market  research  docu¬ 
mentation  needs  to  entail  and  describe  requirements  from 
the  acquisition  team. 

One  source  of  guidance  provided  to  agencies  is  found  in  GAO's 
bid  decision  and  case  studies.  GAO's  publication,  Market  Re¬ 
search:  Better  Documentation  Needed  to  Inform  Future  Procure¬ 
ments  at  Selected  Agencies GAO-15-8,  describes  market  re¬ 
search  as  a  dynamic  process  of  examining  a  marketplace  or 
obtaining  intelligence  about  a  sector.  The  publication,  issued 
late  in  2014,  noted  that  market  research  provides  the  dynam¬ 
ics  "used  to  collect  and  analyze  data  about  capabilities  in  the 
market  that  could  satisfy  an  agency's  procurement  needs." 

Decisions  in  the  federal  court  system,  including  the  Supreme 
Court,  have  clarified  the  role  of  market  research  in  procure¬ 
ment  processes.  In  2015,  the  U.S.  Supreme  Court  ruled  in 
Kingdomware  Technologies  v.  the  United  States  that  market 
research  is  required  in  order  to  understand  products  and  ser¬ 
vices  in  a  particular  market  sector. 

Observations  over  the  last  30  years  indicate  that  many  ac¬ 
quisitions  teams  "put  the  cart  before  the  horse"  when  obtain¬ 
ing  the  outcome  of  a  hypothesis  or  question  about  possible 
sources  for  a  product  or  service.  These  thoughts  transcend 
public  and  private  sector  acquisition  teams  and  lack  value  or 
purpose  in  market  research.  From  the  start,  we  need  to  under¬ 
stand  what  research  entails  and  its  importance. 

In  2017,  the  Merriam-Webster's  Dictionary  noted  that  research 
is  a  planned  exploration  of  materials  and  sources  to  establish 
facts  and  conclusions.  Documentation  of  research  must  in¬ 
clude  methodology,  structure  and  guidance  on  how  to  inter¬ 
pret  the  data.  Without  a  general  framework  or  structure  for 
interpretation,  the  data  collected  are  worthless  to  the  acquisi¬ 
tion  teams  and  customer. 

Furthermore,  in  1992,  The  Journal  of  Marketing  Research  noted 
there  is  a  relationship  between  the  source  and  the  customer  of 
market  research.  All  stakeholders  need  to  be  able  to  trust  the 


outcome  of  the  market  research.  Not  just  the  information,  but 
the  sources  and  interpretation  of  the  whole  process. 

In  2017,  The  Balance  website  (thebalance.com)  noted  that 
the  purpose  of  research  on  business  marketplaces  is  to  nar¬ 
row  results  to  a  specific  target  in  a  market.  Additionally,  in 
2017,  Entrepreneur  magazine  reported  that  market  research 
is  a  dynamic  to  gather,  evaluate,  and  illuminate  market  in¬ 
formation  about  a  product  or  service  for  sale.  Also,  the  term 
"market  intelligence"  has  popped  up  in  the  business  world 
in  connection  with  market  research.  There  is  a  slight  differ¬ 
ence  between  market  intelligence  and  market  research.  In 
2015,  Business  News  Daily  defined  market  intelligence  as  the 
information  obtained  to  make  a  business  determination— i.e., 
acquisition  strategy. 

And  in  2017,  the  Bureau  of  Labor  Statistics  classified  market  re¬ 
search  as  the  process  of  evaluating  market  conditions  concern¬ 
ing  a  probable  need  for  product  or  services  by  a  customer  and 
obtaining  data  from  various  sources.  This  information  provides 
a  clear  understanding  of  vendors'  marketplace  positions  and 
their  pricing  and  ability  to  perform  the  work  required. 

Methodology  and  the  Right  Question 

Now  that  there  is  an  understanding  the  purpose  and  value  of 
researching  a  market  sector  for  products  and  services,  your 
team  needs  the  right  tools  for  acquiring  this  knowledge.  In 
July  2015,  Raconteur's  Future  of  Market  Research  publication 
noted  the  five  critical  steps  to  obtaining  the  best  outcome  of 
researching  a  market  sector:  compute  a  meaningful  topic; 
compile  new  or  old  results;  have  questions  in  any  survey  seek 
more  information  than  the  price;  and  enable  effortless  com¬ 
munication  of  the  results. 

In  2017,  Entrepreneur  magazine  noted  that  an  acquisition  pro¬ 
fessional's  research  of  a  business  sector  would  acquire  primary 
and  secondary  information  or  data.  Primary  data  is  directly  re¬ 
ceived  from  the  source.  This  information  typically  derives  from 
a  Request  for  Information  (RFI)  or  Sources  Sought  (SS),  indus¬ 
try  days  or  questioning  vendors.  Secondary  data  research  as¬ 
sembles  information  from  governmental  organizations— e.g., 
agencies,  trade  associations  and  local  chambers  of  commerce. 

In  developing  the  requirement,  RFI  or  SS  can  be  complicated 
and  a  challenge  for  most  teams.  As  stated  earlier,  GAO's  case 
study  of  DoD's  contracting  of  market  research  was  inadequate 
in  many  ways,  including  its  determination  of  price  reasonable¬ 
ness.  The  typical  technique  for  finding  price  reasonableness  is 
to  look  at  a  vendor's  published  pricing,  historical  data  or  indus¬ 
try  surveys.  For  example,  a  government  survey  can  ask,  "What 
have  you  charged  for  this  product  or  service  in  the  past?" 

At  times,  the  acquisition  teams  forget  to  ask  the  right  ques¬ 
tions.  Initially,  asking  the  right  questions  in  developing  the  mar¬ 
ket  intelligence  is  very  important.  The  topics  should  include 
cost,  historical  experience  information,  technical  information, 
and  management  information. 
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The  following  additional  questions  should  get  your  team 

thinking: 

■  To  what  agency  have  you  provided  the  product  or  service? 

■  After  reviewing  the  RFI,  SS  or  draft  Performance  Work 
Statement  or  Statement  of  Work,  is  your  firm  interested 
in  submitting  a  proposal  to  the  following  Request  for  Pro¬ 
posal  [RFP],  Request  for  Quote  [RFQ],  or  Broad  Agency 
Announcement? 

■  Is  your  firm  interested  in  Prime  or  Subcontractor  work? 


a  vendor's  response  to  a  customer's  need  that  can  affect  or 
improve  performance. 

Results 

Acquisition  professionals  can  use  market  research  results 
or  market  intelligence  in  various  ways  to  arrive  at  a  busi¬ 
ness  decision  for  the  federal  government.  The  data  out¬ 
come  should  represent  a  cross-section  of  a  market  sector. 
The  market  research  processes  should  have  built-in  validity 
and  reliability  methods  to  make  sure  the  data  represents  a 
particular  market  sector.  The  acquisition  team  should  look 


Without  a  general  framework  or  structure  for 
interpretation,  the  data  collected  are  worthless  to  the 
acquisition  teams  and  customer. 


Responsiveness  Versus  Responsibility 

From  my  experience,  acquisition  professionals  use  the  market 
research  process  as  a  "Down  Select''  or  Pre-Source  Selection 
instead  of  obtaining  market  intelligence.  Businessdictionary. 
com  defines  a  "Down  Select"  as  a  reduction  in  sources  as 
the  acquisition  team  proceeds  through  the  process.  Acqui¬ 
sition  professionals  use  methodology  for  determination  of 
responsibility  to  eliminate  a  possible  source  of  a  product  or 
service  that  is  felt  to  lack  sufficient  responsibility.  Procure¬ 
ment  dynamics  of  responsibility  determination  or  FAR  Chap¬ 
ter  9  requirements  need  to  be  completed  by  the  contracting 
officer  and  acquisition  team  later  in  the  process.  This  step 
usually  is  completed  after  proposals  from  an  RFP  or  RFQ  are 
obtained,  which  makes  responsibility  determination  a  part  of 
source  selection. 

In  2011,  GAO  stated  that  the  contracting  officer  only  needs 
to  make  a  determination  of  responsiveness  from  the  mar¬ 
ket  research  tools— e.g.,  RFI,  Industry  Days.  Additionally, 
the  market  intelligence  will  assist  in  deciding  acquisition 
strategies— e.g.,  Small  Business  Set-Asides  (SBSA)  or  Full 
and  Open  Competition.  For  example,  the  respondent  to  a 
RFI  or  SS  need  only  answer  the  intent  of  a  question  to  its 
fullest,  then  the  vendor's  information  should  go  toward  an 
acquisition  strategy,  such  as  SBSA.  The  Cambridge  Dictionary 
defines  responsiveness  by  an  organization  or  individual  to 
a  communication  or  request  as  one  that  is  made  in  a  sat¬ 
isfactory  speedy  manner.  For  this  discussion,  'respondent' 
is  typically  a  business  or  vendor  that  provides  a  service  or 
product  in  a  certain  market.  In  2010,  the  Journal  of  Business 
&  Industrial  Marketing  noted  that  responsiveness  consists  of 


at  each  data  point.  The  market  research  also  needs  to  ad¬ 
dress  limitations  and  delimitations.  A  data  point  could  be 
anomalous  and  not  genuinely  represent  a  market  sector  or 
be  generated  within  a  constantly  changing  industry,  such 
as  that  of  information  technology. 

Another  example  of  understanding  the  market  intelligence 
would  be  the  following:  The  results  or  market  intelligence  from 
an  RFI  or  SS  show  no  responses  or  interest  to  provide  a  service 
or  product.  In  such  a  case,  an  acquisition  team  typically  would 
go  to  a  different  route  or  source.  First,  the  acquisition  team 
should  re-evaluate  the  requirements  package  and  research 
methodology  for  shortcomings,  inconsistencies  and  clarifi¬ 
cations.  The  more  information  provided  to  a  market  sector 
in  market  research  the  better  will  be  the  market  intelligence 
that  a  team  acquires. 

Conclusion 

As  public  servants,  we  need  to  ensure  that  research  and  intel¬ 
ligence  of  the  market  sector  are  done  completely  and  thor¬ 
oughly  as  outlined  in  CICA  and  FAR  10.  Also,  acquisition  pro¬ 
fessionals  need  to  ask  the  right  questions  and  provide  correct 
information  to  the  market  sector  in  order  to  obtain  precise 
results.  Furthermore,  the  market  research  documentation 
should  always  include  a  robust  method,  timeframe,  analysis 
of  the  data,  and  a  recommended  procurement  strategy. 

Passing  on  a  little  knowledge  provides  us  with  purpose  and 
direction. 

The  author  can  be  contacted  at  clcable12@gmail.com. 
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Interdisciplinary 

Competence 

The  Key  to  Exceptional 
Performance 

Carol  J.  Mcllwain,  Ph.D. 


Interdisciplinary 
research ...  integrates 
information,  data, 
techniques,  tools, 
perspectives, 
concepts,  and/or 
theories  from  two  or 
more  disciplines  or 
bodies  of  specialized 
knowledge  to 
advance  fundamental 
understanding  or  to 
solve  problems  whose 
solutions  are  beyond 
the  scope  of  a  single 
discipline  or  area  of 
research  practice. 


—From  "Facilitating 
Interdisciplinary  Research" 
by  the  National  Academy 
of  Sciences,  National 
Academy  of  Engineering, 
and  Institute  of  Medicine 
of  the  National  Academies, 
2005. 


Various  disciplinary  approaches  exist  for 
integrating  knowledge.  Research  dem¬ 
onstrates  that  knowledge  integration  is 
improved  with  interdisciplinary  and  trans- 
disciplinary  approaches  to  problem  solv¬ 
ing.  Integrated  and  interdisciplinary  teams  achieve 
better  problem-solving  skills  by  leveraging  com¬ 
mon  knowledge.  Results  from  academic  institutions 
and  a  3M  Company  study  support  the  development 

Mcllwain  is  a  Defense  Senior  Intelligence  Leader  and  the  director  of  acquisition  and  contracts 
at  the  Office  of  Naval  Intelligence.  She  spent  more  than  33  years  with  the  Navy  serving  in 
program  management  and  engineering  fields.  Her  awards  include  two  Joint  Civilian  Service 
Commendation  awards,  two  Global  War  of  Terrorism  awards,  NATO  medal,  Naval  Air  Warfare 
Center  Innovation  Award  for  Program  Management,  and  the  Daniel  S.  McCauley  Professional 
Achievement  Award  for  Engineering. 
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of  depth  and  breadth  in  disciplinesto  achieve  exceptional 
performance.  Academia  has  established  interdisciplin¬ 
ary  curriculums  and  research  centers  to  facilitate  greater 
advances  of  knowledge  and  technology. 

Solving  complex  problems  requires  different  thinking 
than  finding  solutions  to  simple  problems.  Today's  prob¬ 
lems  are  complex  and  require  a  balancing  of  multiple 
conflicting  or  competing  objectives  and  constraints.  A 
problem  limited  to  a  single  disciplinary  field  is  solvable 
by  experts  in  that  field.  Complex  problems  cross  disci¬ 
plinary  fields  and  require  the  use  of  multiple  disciplines 
to  develop  a  solution.  Integrated  product  teams  (IPTs) 
are  multidisciplinary  and  include  experts  from  several 
functional  areas.  However,  an  IPT  is  challenged  to  fuse 
knowledge  across  disciplines  (Figures  1  and  2). 

An  interdisciplinary  perspective  requires  bridging  knowl¬ 
edge  between  disciplines  to  address  complex  problems. 
Successful  teams  integrate  multiple  disciplines  to  frame 


a  problem,  agree  on  a  methodological  approach,  and 
collaboratively  analyze  data.  Exceptional  teams  do  a  bet¬ 
ter  job  of  integrating  knowledge. 

Through  cross-disciplinary,  interdisciplinary  or  trans- 
disciplinary  concepts,  the  acquisition  workforce  devel¬ 
ops  a  systemic  view  and  problem-solving  skills  using 
fused  knowledge  and  can  develop  a  multiple  disciplin¬ 
ary  understanding.  Greater  integration  of  disciplinary 
knowledge  enables  the  development  of  more  effective 
critical  thinking  and  innovative  ideas  than  are  possible 
in  traditional  multidisciplinary  teams. 

An  intradisciplinary  approach  relates  to  a  single  disci¬ 
pline.  A  multidisciplinary  approach  is  an  integrated  team 
that  gains  multiple  views  from  members  grounded  in 
different  disciplines.  Cross-disciplinary  views  one  disci¬ 
pline  from  the  perspective  of  others,  which  is  sometimes 
described  as  akin  to  looking  through  a  lens.  Interdisci¬ 
plinary  approaches  use  synthesis  to  integrate  knowledge 
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Figure  1.  Separate  Areas  ot  Knowledge  A  discipline  is  a  specific  field  of  learn¬ 

ing  or  body  of  knowledge  with  defined 
elements  such  as  phenomena,  assump¬ 
tions,  epistemology,  concepts,  theories 
and  methods  that  distinguish  it  from 
other  fields  or  bodies  of  knowledge.  The 
pursuit  of  further  knowledge  and  explo¬ 
ration  typically  deals  in  depth  within  one 
field  to  gain  further  understanding.  The 
concept  of  interdisciplinary  studies  re¬ 
quires  not  only  depth  but  also  breadth 
across  one  or  more  disciplines  to  un¬ 
derstand  the  integration  of  knowledge 
between  disciplines.  The  term  "career 
fields  "describes  areas  of  acquisition  cat¬ 
egories  such  as  budgeting,  engineering, 
logistics,  contracting,  manufacturing, 
test  and  evaluation,  etc.  In  Department 
of  Defense  (DoD)  acquisition,  functional 
areas  are  synonymous  with  career  fields 
that  define  knowledge  areas. 


Being  a  trained  expert  is  a  disadvantage  in 
some  situations;  competence  bias  hinders 
thinking  beyond  that  single  view.  People 
educated  in  multiple  disciplines  are  better 
able  to  design  and  apply  a  process  based 
on  certain  conditions  and  constraints. 


Source  of  figures:  Department  of  Defense 
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Figure  2.  Interdisciplinary  Fusing 
of  Knowledge 


from  different  disciplines  (Figure  3).  A  transdisciplinary  ap¬ 
proach  integrates  to  the  extent  of  producing  a  new  discipline 
that  provides  insight  into  an  area  not  previously  understood. 
Examples  include  biochemistry,  biomedical,  political  ecology, 
educational  psychology,  and  neuropsychology. 


This  produces  flexible  thinking  that  challenges  trained 
specialists.  Using  a  multidisciplinary  approach  through 
a  team  of  disciplinary  or  functional  experts  does  not 
achieve  integration  or  synthesis  of  knowledge  because 
there  is  no  common  ground.  A  common  ground  provides 
the  linkage  between  disciplines  and  creates  insight  and  an 
ability  to  gain  multiple  perspectives  and  use  knowledge 
in  multiple  applications. 

The  interdisciplinary  knowledge  requires  that  planners 
know  at  least  two  discipline  areas  in  order  to  be  able  to 
establish  linkages  across  those  areas.  In  a  knowledge 
sphere,  nodes  are  the  knowledge  base  and  linkages  create 
insight  between  knowledge  bases.  The  common  ground 
attained  through  linkages  provides  the  means  to  gain  in¬ 
sight  between  disciplines. 

Teams  of  experts  produce  a  multidisciplinary  approach, 
viewing  a  problem  from  their  own  discipline  and  recom¬ 
mending  solutions  based  on  their  particular  areas  of  ex¬ 
pertise.  An  integrated  team  lead  either  selects  one  solu¬ 
tion  or  needs  to  merge  the  multiple  solutions  into  a  single 
fused  solution,  requiring  an  interdisciplinary  approach  and 
accompanying  knowledge  of  the  various  functional  areas 
to  develop  a  single,  comprehensive  solution.  A  fused  solution 
is  different  from  any  single  functional  solution. 

The  team  lead  typically  lacks  interdisciplinary  knowledge, 
selecting  the  one  solution  that  appears  to  offer  the  most 
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Figure  3.  Interdisciplinary 
Knowledge  Integration 


advantages  for  one  functional  area  with  fewer  disadvan¬ 
tages  for  the  other  areas.  This  is  not  an  integrated  solution. 
A  true  IPT  lead  would  have  an  interdisciplinary  background 
with  knowledge  and  experience  across  pertinent  areas. 
Therefore,  program  managers  benefit  from  interdisciplin¬ 
ary  competence. 

Jay  Forrester  of  the  Massachusetts  Institute  of  Technology 
developed  the  system  dynamic  concept  as  a  theoretical  ap¬ 
proach  to  understanding  complex  systems.  He  initially  devel¬ 
oped  the  tool  in  the  engineering  domain  but  later  applied  it 
to  the  business  world.  The  system  dynamics  paradigm  con¬ 
cluded  that  decisions  produce  disappointing  results  because 
important  casual  relationships  are  overlooked  or  misread— 
usually  by  assuming  a  linear  or  unidirectional  relationship 
versus  a  nonlinear  and  multidirectional  relationship.  Apply¬ 
ing  an  engineering  systems  perspective  to  business  opera¬ 
tions  is  an  interdisciplinary  approach.  Specialist-  or  expert- 
dominated  organizations  often  simplify  problems  and  reduce 
them  to  linear,  unidirectional  casual  relationships,  even  if  the 
problem  is  more  complex  and  multidirectional.  A  simplified 
problem  can  lead  to  solving  the  wrong  problem.  Research 
laboratories  recognize  the  need  to  solve  complex  problems 
with  complex  solutions. 

An  example  of  an  interdisciplinary  application  is  the  National 
Synchrotron  Light  Source  (NSLS)  at  Brookhaven  National 
Laboratory  of  Upton,  New  York,  on  Long  Island.  The  device 
produces  synchrotron  radiation,  or  light  produced  by  charged 
particles  bending  in  a  magnetic  field.  Although  initially  estab¬ 
lished  to  support  physics-related  research,  NSLS  has  evolved 
into  an  interdisciplinary  laboratory: 


■  It  is  a  single  facility  that  supports  multiple  projects  in  dif¬ 
ferent  fields. 

■  Each  project,  large  or  small,  requires  integration  of  knowl¬ 
edge,  techniques  and  perspectives  from  several  disci¬ 
plines  or  specialized  fields. 

■  A  few  of  the  projects  are  creating  new  disciplines. 

■  Many  projects  require  more  than  a  single  instrument  and 
incorporate  knowledge,  techniques  and  perspectives  from 
additional  facilities  at  Brookhaven  and  other  laboratories. 

Common  ground  for  integrating  disciplines  is  attainable 
through  mathematics  and  statistics,  which  cross  many 
disciplines,  including  natural  and  physical  sciences,  social 
sciences,  humanities,  and  applied  professions.  The  interdis¬ 
ciplinary  field  of  survey  research  merged  the  fields  of  soci¬ 
ology  and  political  science  and  created  specialists  in  data 
collection.  Computers  transformed  areas  of  multivariate 
data  analysis  and  mathematical  modeling  as  a  foundation 
for  operational  research  across  multiple  areas:  economics, 
sociology,  political  science,  engineering,  business  and  mili¬ 
tary  operational  planning. 

Game  theory  merged  economics  and  psychology,  evaluat¬ 
ing  strategies  based  on  predicted  behavior,  and  has  become 
foundational  for  strategic  negotiations  in  business,  political 
science,  international  relations,  and  military  operational  plan¬ 
ning.  Mathematical  modeling,  game  theory  and  statistical 
analysis  are  tools  to  create  common  ground  for  interdisci¬ 
plinary  learning. 

A  joint  effort  by  the  National  Academy  of  Sciences,  National 
Academy  of  Engineers,  and  the  Institute  of  Medicine's  Com¬ 
mittee  on  Facilitating  Interdisciplinary  Research  and  Com¬ 
mittee  on  Science,  Engineering,  and  Public  Policy  studied 
the  concept  of  interdisciplinary  research  and  published  the 
results  in  the  book,  Facilitating  Interdisciplinary  Research ,  to 
document  the  benefits  that  interdisciplinary  studies  provide 
for  academic  understanding. 

Although  the  findings  focused  on  academic  application  for 
universities,  institutes,  and  laboratories,  they  are  easily  trans¬ 
ferable  to  government  organizations  and  industry  efforts  to 
solve  problems  and  improve  the  management  of  projects  and 
programs.  Complexity  theory  yields  nonlinear  results,  creating 
the  need  for  interdisciplinary  approaches  and  crossing  disci¬ 
plines  to  leverage  multiple  benefits.  Defense  acquisition  ben¬ 
efits  from  this  objective,  leveraging  initiatives  across  multiple 
functional  areas  fortechnology  development  and  achievement 
of  nonlinear  results. 

A  workforce  skilled  in  single  disciplines  challenges  the  integra¬ 
tion  of  complex  technology  development.  Defense  acquisition 
benefits  from  an  interdisciplinary  approach  led  through  sys¬ 
tems  engineering  and  program  management.  An  interdisci¬ 
plinary  workforce  facilitates  technology  development,  first  by 
integrating  the  individual's  knowledge  within  him-  or  herself 
and  then  integrating  individual  knowledge  across  the  team. 
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Unconscious  bias  limits 
recognizing  multiple  solutions, 
resulting  in  a  reversion  to  the 
current  knowledge  base  without 


pursuing  further  information. 


Systems  engineering  facilitates  common  ground  across  de¬ 
velopment  disciplines,  and  program  management  facilitates 
common  ground  across  an  entire  acquisition  team. 

When  someone  is  involved  in  innovation,  existing  brain  con¬ 
nections  (neurons)  must  significantly  change  and  cross  wider 
areas  of  the  brain  dealing  with  different  types  of  knowledge 
and  problems  in  order  to  assimilate  very  different  concepts 
and  challenge  long-held  assumptions.  Tara  Swart's  book, 
Neuroscience  of  Leadership,  states  that  "working  at  the  inter¬ 
stices  of  domains  of  knowledge  can  also  create  dissonance, 
and  being  engaged  and  interested  in  a  wider  range  of  ac¬ 
tivities  than  just  one's  own  domain  is  also  often  a  mark  of 
exceptionally  creative  people."  The  strengthening  of  neurons 
in  the  brain  creates  bias  and  limits  problem  solving.  Uncon¬ 
scious  bias  limits  recognizing  multiple  solutions,  resulting  in 
a  reversion  to  the  current  knowledge  base  without  pursuing 
further  information. 

Interdisciplinary  Results — 3M  Company 
Study  on  Innovation 

Innovation  relies  on  an  individual's  expertise  to  generate  new 
knowledge  or  create  new  ideas  through  recombining  ideas  to 
create  innovative  applications.  In  the  research  paper,  "Bal¬ 
ancing  Breadth  and  Depth  of  Expertise  for  Innovation:  A  3M 
Story,"  the  authors  state: 

Even  though  many  inventions  are  created  when  individuals 
work  in  teams,  studies  allude  to  the  observation  that  individu¬ 
als  are  effective  in  combining  existing  knowledge  to  generate 
new  knowledge  and  innovations.  Innovative  ideas  and  insights 
first  occur  to  individuals,  before  such  ideas  are  subsequently 
shared  at  the  group  levels  and  institutionalized  at  the  organiza¬ 
tional  level.  Fundamentally,  this  highlights  that  individuals  are 
the  basic  unit  in  which  knowledge  integration  and  knowledge 


creation  takes  place,  regardless  of  whether  individuals  work 
alone  or  in  teams. 

If  innovative  ideas  are  not  created  at  the  unit  level,  they  will 
not  be  created  at  the  team  level. 

A  study  of  how  inventors'  breadth  and  depth  of  expertise  influ¬ 
ence  innovation  at  3M  exceeded  previous  research  focused  on 
a  single  indicator— technical  success  achieved  by  the  inventor. 
The  3M  study  examined  three  indicators: 

■  The  number  of  inventions  generated. 

■  The  extent  to  which  the  inventor  has  a  significant  impact 
on  the  technical  domain. 

■  The  inventor's  career  success,  in  terms  of  commercial 
value  brought  by  converting  the  inventions  into  products 
that  generate  sales  for  commercial  organizations. 

The  study  concluded  that  generalists  (breadth)  create  many 
inventions  that  are  not  technically  influential;  specialists 
(depth)  create  fewer  inventions  but  these  are  technically  in¬ 
fluential.  The  combination  of  breadth  and  depth  (polymath) 
of  expertise  creates  the  most  valuable  inventors  that  have 
established  a  record  for  effectively  converting  inventions  into 
commercially  successful  products.  In  other  words,  the  poly¬ 
math  earned  the  most  money  for  3M  by  producing  the  most 
marketable  inventions. 

A  "specialist"  achieves  very  deep  knowledge  through  study 
and  experience.  Studies  found  that  specialists  acquire  ability 
for  detailed  and  accurate  analysis  that  lead  to  solutions  of  dif¬ 
ficult  technical  problems  in  their  areas  of  expertise.  Specialists 
also  make  difficult  trade-offs,  and  through  deep  knowledge 
can  better  predict  what  will  go  wrong.  They  create  ground¬ 
breaking  innovations  by  persistently  exploring  more  deeply 
in  a  particular  area. 

Generalists  have  knowledge  in  a  broad  range  of  areas  but  lack 
expertise  in  any  particular  area.  Generalists  tend  to  enjoy  new 
work  and  become  bored  when  confined  to  one  area;  this  inhib¬ 
its  their  ability  to  develop  the  specialist's  depth.  Generalists 
focus  on  the  application  of  technologies  to  useful  products  and 
integration  of  multiple  technologies  into  a  product,  creating 
innovation  through  a  broader  focus. 

Polymaths  have  acquired  significant  depth  and  breadth  by 
first  becoming  experts  in  one  area  and  then  expanding  their 
expertise  into  other  areas.  One  polymath  inventor  at  3M  had 
described  the  benefits  of  both  in  the  study:  "his  depth  of  ex¬ 
pertise  plays  a  key  role  in  identifying  the  technical  contribu¬ 
tions  of  an  idea,  while  he  draws  upon  a  breadth  of  expertise 
to  evaluate  the  potential  ways  the  invention  can  impact  dif¬ 
ferent  industries."  By  balancing  the  combination  of  depth  and 
breadth,  polymath  inventors  become  astute  at  applying,  inte¬ 
grating  and  recombining  technology  of  their  domains  across 
other  technologies  and  applications.  Generalist  inventors 
focus  on  finding  new  applications  for  a  developed  technology 
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Figure  4.  Lack  of  Linkage  Between  Integrated  Functions 


but  lack  the  depth  to  develop  a  new  technology.  Specialists 
develop  the  technology  but  lack  the  breadth  needed  to  adapt 
it  to  various  applications. 

How  are  polymaths  developed?  The  acquisition  of  depth 
probably  precedes  the  acquisition  of  breadth.  Once  depth 
is  acquired,  the  polymath  can  use  that  learning  to  acceler¬ 
ate  the  achievement  of  depth  in  other  areas— acquiring  the 
ability  to  go  deep  and  then  applying  that  ability  to  go  broad. 
When  breadth  is  established  without  first  acquiring  depth, 
depth  probably  will  never  be  attained. 

The  study  concluded  that  organizations  need  specialists, 
generalists,  and  polymaths  but  that  "both  breadth  and 
depth  of  expertise  are  required  to  effectively  convert  in¬ 
ventions  into  commercially  successful  products  that  bring 
sales  and  value  to  the  organization.  The  polymaths  contrib¬ 
uted  not  only  by  generating  inventions,  but  applying  these 
inventions  widely  to  multiple  parts  of  the  organization,  in¬ 
tegrating  with  multiple  technologies,  thus  becoming  the 
most  valued  scientists  of  3M." 

This  combination  is  created  through  starting  careers  that 
go  into  significant  depth  in  single  areas.  Over  time,  signifi¬ 
cant  knowledge  and  experience  outside  that  one  domain  is 
acquired.  By  leveraging  an  understanding  about  how  one 
becomes  an  expert,  expertise  is  developed  more  quickly  in 
other  areas. 

A  polymath  develops  an  interdisciplinary  perspective  by  at¬ 
taining  depth  and  breadth  across  multiple  disciplines,  lever¬ 
aging  the  knowledge  interface  between  functional  areas  to 


develop  the  interdisciplinary  perspec¬ 
tive  faster. 

Rockefeller  University 

Rockefeller  University  (formerly  the 
Rockefeller  Institute)  in  Manhattan  has 
implemented  a  unique  approach  to  bio¬ 
medicine  research.  The  university  holds 
more  major  discoveries  in  biomedicine 
than  any  other  such  research  institute. 
Its  faculty,  fellows  and  alumni  have  in¬ 
cluded  27  Nobel  laureates  in  physiology 
or  medicine  and  chemistry,  22  Lasker 
Award  recipients,  five  faculty  members 
named  as  MacArthur  fellows,  20  recip¬ 
ients  of  the  National  Medal  of  Science, 
and  18  National  Academy  of  Medicine 
members.  The  joint  National  Academy 
of  Sciences  study  said  of  the  university 
that  "major  discoveries  occurred  relat- 
edly  because  there  was  a  high  degree 
of  interdisciplinary  and  integrated  ac¬ 
tivity  across  diverse  fields  of  science, 
and  because  of  leadership  that  gave 
particular  attention  to  the  creation  and 
maintenance  of  a  nurturing  environment,  though  with  rigorous 
standards  of  scientific  excellence ...  there  are  three  important 
characteristics:  a  high  level  of  scientific  diversity,  low  levels  of 
internal  differentiation,  and  visionary  leadership."  The  univer¬ 
sity  has  13  interdisciplinary  centers  dedicated  to  investigating 
the  interstices  between  general  areas  of  study,  such  as  physics 
and  biology  or  biochemistry  and  structural  biology.  Other  cen¬ 
ters  focus  on  multidisciplinary  methods  of  addressing  specific 
biomedical  problems. 

This  performance  was  achieved  by  recruiting  researchers 
with  diverse  scientific  and  cultural  backgrounds,  most  of 
them  working  in  fields  that  crossed  disciplines.  The  uni¬ 
versity  did  not  organize  around  academic  disciplines  but 
instead  around  a  laboratory  environment  deemed  "without 
walls"  to  promote  cross-knowledge  utilization  of  scientists 
on  research  projects. 

The  Army's  Campaign  Planning  Handbook  recognizes  a  multi¬ 
disciplinary  perspective  for  solving  complex  problems  in  the 
counterinsurgency  mission.  A  system-of-systems  concept 
overlaps  the  six  areas  of  political,  military,  economic,  social,  in¬ 
formation  and  infrastructure  and  aligns  with  those  disciplines. 
An  interdisciplinary  approach  integrates  the  knowledge  across 
these  functional  areas. 

For  teams  to  be  effective,  team  members  need  common 
ground  in  order  to  develop  fused  ideas.  Greater  depth  is  de¬ 
veloped  in  each  discipline  through  linkages  of  the  knowledge 
nodes.  A  team's  work  likely  will  evaluate  solutions  based  on 
a  particular  functional  approach  when  there  is  little  common 
ground  (few  linkages)  between  the  functions  (Figure  4). 
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An  interdisciplinary  team  merges  knowledge  across  multiple 
disciplines;  each  team  member's  knowledge  crosses  at  least 
two  disciplines  (Figure  5).  For  complex  problems,  greater 
insight  is  gained  through  crossing  disciplines.  The  need  to 
acquire  breadth,  even  when  pursuing  advanced  degrees,  is 
recognized  in  recommendations  for  a  new  vision  on  the  part  of 
the  academic  institutional  structure.  As  cited  by  the  National 
Academy  of  Sciences: 


DoD's  complex  technological  ad¬ 
vancement  requires  a  paradigm  shift 
from  previous  knowledge-based  train¬ 
ing  and  learning.  An  interdisciplinary 
program  manager  can  facilitate  the  integration  of  a  team's 
knowledge.  In  the  case  of  both  societal  problem  solving  and 
technological  advancement  problem  solving,  it  has  been 
proved  that  the  interdisciplinary  approach  enables  excep¬ 
tional  performance  unattainable  through  a  single  disciplinary 
or  multidisciplinary  approach.  & 


The  author  can  be  contacted  at  carol.mcilwain@navy.mil. 


Figure  5.  Knowledge  Merger  Among  Team  Members 


Promoting  development  of  a  primary 
field  for  the  workforce  establishes 


expertise— and  then  secondary  field 
certifications  develop  breadth.  Inte¬ 
grated  interdisciplinary  teams  lever¬ 
age  the  connections  of  knowledge  and 
provide  a  means  for  "seeing  the  space 
between  nodes  of  knowledge."  Com¬ 
mon  ground  connects  two  different 
areas  sharing  modeling  or  statistical 
tools;  analytical  tools  should  comple¬ 
ment  training  curriculums  and  posi¬ 
tion  assignments.  Interdisciplinary 
individuals  resolve  complex  problems 
across  multiple  disciplines  through  the 
internal  fusion  of  knowledge  and  un¬ 
derstanding.  Common  ground  is  de¬ 
veloped  through  linkages  established 
between  functional  areas. 


A  matrix  structure  in  a  university  might  include  many  joint  fac¬ 
ulty  appointments  and  Ph.D.s  granted  in  more  than  one  depart¬ 
ment  which  would  enable  participants  to  address  cross-cutting 
questions  more  easily.  It  might  create  numerous  interdisciplin¬ 
ary  courses  for  undergraduates,  provide  mentors  who  bridge 
the  pertinent  disciplines,  and  equally  important,  offer  faculty 
numerous  opportunities  for  continuing  education  whereby  they 
could  add  both  depth  and  breadth  of  knowledge  throughout 
their  careers. 

Implementation  of  interdisciplinary  competence  requires 
changes  in  current  management  recruitment,  retention  and 
promotion  policies  within  the  human  resource  management 
area.  Position  selections  require  criteria  that  balance  specific 
skills  for  the  current  position  with  the  ability  to  grow  beyond 
that  position  through  a  breadth  of  knowledge  and  experience. 
Acquisition  development  requires  special  recruitment  policies 
to  target  polymath  individuals  for  certain  senior  positions.  In¬ 
centives  are  needed  in  recruitment  and  selection  to  attract  the 
best  individuals  for  key  positions.  This  means  that,  for  vertical 
and  horizontal  development  and  advancement,  a  paradigm 
shift  is  needed  away  from  traditional  thinking.  Expertise  in 
acquisition  policy  should  merely  complement  one's  primary 
functional  expertise. 


MDAP/MAIS  Program 
Manager  Changes 


With  the  assistance  of  the  Office  of  the  Secretary  of 
Defense,  Defense  AT&L  magazine  publishes  the  names 
of  incoming  and  outgoing  program  managers  for  major 
defense  acquisition  programs  (MDAPs)  and  major  au¬ 
tomated  information  system  (MAIS)  programs.  This  an¬ 
nouncement  lists  recent  such  changes  of  leadership  for 
both  civilian  and  military  program  managers. 

Navy/Marine  Corps 

CAPT  Christopher  DeSena  relieved  CAPT  Joseph  Kan 

as  program  manager  for  the  Mobile  User  Objective  Sys¬ 
tem  (PMW  146)  on  Sept.  1,  2017. 

CAPT  Andrew  Gibbons  relieved  CAPT  Mark  Glover 

as  program  manager  for  the  Navy  Multi  Band  Terminal 
(PMW  170)  on  Nov.  16,  2017. 
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Cybersecurity  is  a  team  sport  that  requires  Program  Management,  Cyber/ 
Information  Technology,  Engineering,  Test  and  Evaluation,  Finance,  Lo¬ 
gisticians  and  Contracting.  In  order  to  improve  the  survivability  of  our 
Department  of  Defense  (DoD)  systems  under  cyberattack,  we  must 
consider  cybersecurity  in  the  earliest  phases  of  contract  planning— from 
acquisition  planning  to  contract  maintenance  and  closeout. 

If  cybersecurity  isn't  properly  integrated  into  the  solicitation  process  we  won't  (1)  know  if  the  offerors  are  capable 
of  delivering  our  cybersecurity  requirements,  (2)  be  able  to  discriminate  between  offeror  proposals  or  (3)  be  able 


Kendall  and  Long  are  professors ,  respectively,  of  cybersecurity  and  contract  management  at  the  Defense  Acquisition  University's  South  Region 
in  Huntsville ,  Alabama.  Kendall ,  a  retired  Air  Force  colonel,  is  a  former  deputy  division  chief  for  Information  Technology/Cyber  Programs,  Air 
Staff  Information  Dominance  Directorate,  Office  of  the  Assistant  Secretary  of  the  Air  Force  for  Acquisition.  Long  also  performs  consulting 
efforts  for  the  Department  of  Defense  and  other  federal  agencies  and  participates  as  a  subject-matter  expert  ensuring  curriculum  currency 
and  enhancing  processes  within  the  contracting  career  field.  He  is  the  course  manager  for  DAU's  Contingency  Contracting  Course,  CON  234. 
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Figure  1.  Contracting  Touchpoints  Across  the  Acquisition  Life  Cycle 
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Key  to  Figure:  ICD= I nitial  Capabilities  Document;  CDD=Capability  Development  Document;  CDD-V=Capability  Development  Docu¬ 
ment  Validation;  CPD=Capability  Production  Document;  CDR=Critical  Design  Review;  DRFPRD=Development  Request  for  Proposals 
Release  Decision;  FOC=  Ful I  Operational  Capability;  FRPDR=Full-Rate  Production  Decision  Review;  LRIP=Low-Rate  Initial  Production; 
MDD=Materiel  Development  Decision;  PDR=Preliminary  Design  Review;  RFI  =  Request  for  Information;  RFP=Request  for  Proposal 

Source:  Adapted  by  authors  from  DAU's  Cybersecurity  and  Acquisition  Life-cycle  Integration  Tool 


to  provide  the  proper  oversight  since  we  may  not  have  asked 
for  the  appropriate  data  to  monitor  contract  performance. 
Ensuring  cybersecurity  is  appropriately  addressed  in  the  so¬ 
licitation  process  involves  more  than  selecting  Federal  Acquisi¬ 
tion  Regulation  (FAR)-Defense  Federal  Acquisition  Regulation 
Supplement  (DFARS)  clauses! 

Cybersecurity  requirements,  like  other  system  requirements, 
underpin  the  solicitation  process.  Early  involvement  by  the 
contracting  officer  is  the  key  to  successful  incorporation  of  cy¬ 
bersecurity  requirements  into  the  Request  for  Proposal  (RFP), 
source  selection  and  post-award  contractor  execution  activi¬ 
ties.  Additionally,  contracting  officers  need  to  understand  a 
program's  cybersecurity  requirements  and  risks  to  inform  con¬ 
tract  type  selection.  Figure 
1  shows  touch  points  in  the 
life  cycle  where  contract¬ 
ing  solicitation  activities 
should  include  cybersecu¬ 
rity  considerations. 

Understanding  and 
Communicating 
Requirements 

Contracting  for  cybersecu¬ 
rity  begins  in  the  Require¬ 
ments  Phase.  It  is  impera¬ 
tive  that  the  contracting 
officer  understand  the  pro¬ 
gram's  cybersecurity  re¬ 
quirements  and  construct 
a  contracting  strategy  to 
determine  whether  offer¬ 
ors  are  capable  of  deliver¬ 
ing  those  requirements. 


Many  cybersecurity  requirements  are  included  in  the  man¬ 
datory  System  Survivability  Key  Performance  Parameter 
(KPP)  because  Cyber  Survivability  is  now  a  key  element.  All 
cybersecurity-required  capabilities  (including  those  derived 
from  the  Risk  Management  Framework  [RMF]  process)  are 
decomposed  into  the  government-owned  technical  require¬ 
ments  baseline.  Traceability  and  balance  between  cyberse¬ 
curity  requirements,  security  controls  and  mission  needs  is 
of  critical  importance.  This  is  where  the  contracting  officer 
can  help  the  program  manager  (PM)  make  informed  trade- 
space  decisions. 

Cybersecurity  requirements  should  be  communicated  with 
industry  through  various  forums  (e.g.,  Industry  Days,  Sources 


Figure  2.  Putting  Cybersecurity  Requirements  on  Contract 
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Sought  Synopsis,  Request  for  Information  (RFI),  one-on-one 
meetings,  Draft  RFP,  Preproposal  Conferences,  etc.)  and  ulti¬ 
mately  included  in  the  final  RFP.  This  will  provide  industry  with 
a  better  understanding  of  the  breadth  and  depth  of  cyberse¬ 
curity  requirements.  See  Figure  2. 

Source  Selection 

Clearly  communicated  cybersecurity  requirements  provide 
potential  offerors  information  on  which  to  base  their  pro¬ 
posed  solutions  and  provide  DoD  with  measures  to  evaluate 
offeror  capability  and  solutions.  Cybersecurity  risk  should 
be  a  consideration  when  determining  evaluation  criteria  to 
provide  discriminators  among  proposals.  The  following  are 
just  a  few  resources  providing  examples  of  cybersecurity 
considerations  that  can  be  incorporated  into  the  RFP:  the 


Additional  cybersecurity-related  DFARS  clauses  include: 

■  DFARS  Clause  252.204-7008— Compliance  with  Safe¬ 
guarding  Covered  Defense  Information  Controls 

■  DFARS  Clause  252.204-7009— Limitations  on  the  Use 
or  Disclosure  of  Third-Party  Contractor  Reported  Cyber 
Incident  Information 

■  DFARS  Clause  252.239-7009— Representation  of  Use  of 
Cloud  Computing 

■  DFARS  Clause  252.239-7010— Cloud  Computing  Services 

■  DFARS  Clause  252.239-7017— Notice  of  Supply  Chain 
Risk 

■  DFARS  Clause  252.239-7018— Supply  Chain  Risk 

The  foregoing  is  not  an  all-inclusive,  one-size-fits-all  list,  and 


...We  need  to  incentivize  contractor  efforts  beyond  "check  the 
box"  minimum  performance  by  incorporating  specific  incentives 
designed  to  encourage  exceptional  performance.* 
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DoD  Program  Manager's  Guidebook  for  Integrating  the  Cy¬ 
bersecurity  Risk  Management  Framework  (RMF)  into  the 
System  Acquisition  Lifecycle;  the  Guide  for  Integrating  Sys¬ 
tems  Engineering  into  DoD  Acquisition  Contracts;  Suggested 
Language  to  Incorporate  System  Security  Engineering  for 
Trusted  Systems  and  Networks  into  DoD  Requests  for  Pro¬ 
posals;  and  https://shortcut.dau.mil/ncma/cyber_contracts. 
Table  1  is  a  sampling  of  these  considerations. 

FAR/DFARS  Clauses  and  Public  Law 

The  procurement  team  should  work  together,  but  the  con¬ 
tracting  officer  has  the  ultimate  responsibility  for  FAR  and 
the  Defense  FAR  Supplement  (DFARS)  requirements.  DoD 
Instruction  (DoDI)  5000.02,  Change  3,  Enclosure  14,  specifi¬ 
cally  calls  out  the  following: 

■  FAR  Clause  52.204-2  Security  Requirements 

■  FAR  Clause  52.204-21  Basic  Safeguarding  of  Covered 
Contractor  Information  Systems 

■  Section  933,  National  Defense  Authorization  Act,  FY 
[Fiscal  Year]  2013,  Public  Law  112-239— Improvements 
in  Assurance  of  Computer  Software  Procured  by  the 
Department  of  Defense 

■  DFARS  Clause  252.204-7012  Safeguarding  Covered  De¬ 
fense  Information  and  Cyber  Incident  Reporting 

■  Section  937,  National  Defense  Authorization  Act,  FY 
2013,  Public  Law  113-66— Joint  Federated  Center  for 
Trusted  Defense  Systems  for  the  Department  of  Defense 


contracts  should  be  based  on  individual  program  requirements 
and  risk! 

Effective  Cybersecurity  Government  Oversight 

To  determine  if  cybersecurity  requirements  are  being  imple¬ 
mented  effectively,  the  right  data  and  tools  need  to  be  written 
into  the  contract.  The  following  are  examples  of  data,  artifacts 
and/or  activities  that  we  might  monitor: 

■  Software  vulnerability  scans  (static  and  dynamic) 

■  Formal  code  inspections 

■  Software  quality  measures  and  configuration  control 

■  Test  coverage 

Incentivize  Cybersecurity  Performance 

Incentives  are  fundamental  elements  of  any  contract.  The 
contract  itself  motivates  successful  performance  from  a  mon¬ 
etary  standpoint,  future  relevant  work  and  "brand"  reputation. 
However,  since  cybersecurity  historically  has  been  treated  as 
a  compliance  checklist,  perhaps  we  need  to  incentivize  con¬ 
tractor  efforts  beyond  "check  the  box"  minimum  performance 
by  incorporating  specific  incentives  designed  to  encourage 
exceptional  performance.  In  the  face  of  ever-increasing  cyber 
threats,  cybersecurity  may  be  a  critical  risk  area  necessitating 
extra  effort  to  mitigate  those  risks. 

There  can  be  a  combination  of  financial  and  nonfinancial  in¬ 
centives,  including  improved  cash  flow,  increased  business 
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Table  1.  Request  for  Proposal  (RFP)  Sample  Cybersecurity  Considerations 


Request  for  Proposal 


Section  B 

Supplies  or  services  and  prices/costs 

•  Review  all  CDRL  deliverables  for  inclusion  of  cybersecurity  execution  support  (e.g.,  data  rights,  test  data,  test  plans, 
source  code  deliveries,  prototype  quantity,  and  delivery  times  and/or  locations). 

Section  C 

Description/Specification/Statement  of  Work 

■  State— in  performance-based  terms— cybersecurity  requirements  levied  on  the  contractor. 

•  Include  cybersecurity  system/technical  requirements  in  the  SRD/TRD. 

•  Identify  the  system  RMF  categorization,  overlays,  RMF  security  controls  to  inform  scope. 

■  Identify  any  specific  design,  contractor  testing  or  artifacts  that  enable  compliance  with  cybersecurity  requirements. 

Section  E 

Inspection  and  acceptance 

•  Ensure  that  a  quality  assurance  surveillance  plan  exists  to  monitor  contractor  performance,  including  cybersecurity. 

Section  F 

Deliveries  or  performance 

•  Ensure  that  cybersecurity-related  items  are  addressed  like  any  other  type  of  requirement  (e.g.,  test  article  delivery, 
contractor  support  for  repair,  etc.). 

Section  H 

Special  contract  requirements 

•  List  applicable  cybersecurity  special  contract  requirements  (e.g.,  handling  of  data,  software  license  management  and 
maintenance,  use  of  contractor  facilities  for  cybersecurity  testing). 

Seciton  1 

Contract  clauses 

•  Cybersecurity-specific  contract  clauses  should  be  considered. 

Section  J 

List  of  attachments 

•  Consider  applicable  cybersecurity  attachments  (e.g.,  a  DoD  component  RMF  Guide,  Program  Protection  Plan). 

Section  K 

Representations,  Certifications,  and  Other  Statements  of  Offerors  or  Respondents 

■  Include  requests  for  certification  that  support  the  cybersecurity  strategy  (e.g.,  National  Security  Agency 
certifications  of  cryptographic  algorithms  or  equipment,  and  certification  of  cross  domain  solutions). 

Section  L 

Instructions,  Conditions,  and  Notices  to  Offerors  or  Respondents 

■  Describe  the  experience  of  cybersecurity  staff,  predicted  staffing  levels,  and  the  application  of  cybersecurity  best 
practices  and  its  alignment  with  the  contractor  management  structures  for  SSE  and  T&E. 

■  Define  the  contractor's  responsibilities  for  cybersecurity  and  the  alignment  of  those  responsibilities  in  contrast  to  the 
government  for  required  SSE  and  T&E  activites  (e.g.,  contractor  cybersecurity  testing,  developmental  testing,  and 
integrated  testing). 

•  Describe  the  contractor's  approach  for  technical  data,  including  management,  ownership,  control,  timely  access,  and 
delivery  of  all  cybersecurity  data,  including  raw  test  data,  to  support  the  evolving  technical  baseline. 

■  Define  CDRLs  and  select  applicable  DIDs.  Identify  any  cybersecurity-related  data  products  contractors  must  provide. 

■  Describe  contractor's  approach  for  satisfying  the  Program  Protection  Plan. 

•  Describe  contractor's  approach  for  detecting  counterfeit  components  and  use  of  cyber-certified  products  for  hard¬ 
ware  and  software. 

•  Describe  the  contractor's  access  to  government  cyber  ranges,  use  of  commercial  and/or  government  Blue  and/or  Red 
teams  during  cybersecurity  testing. 

Section  M 

Evaluation  Factors  for  Award 

•  Prior  performance  in  integrating  cybersecurity  considerations  into  the  program's  SE,  SSE  and  T&E  processes. 

■  Meet  cybersecurity  workforce  certification  and  training  requirements  in  DoDD  8140.01  and  DoD  8570. 01-M,  and 
investigative  requirements  per  DoDI  8500.01. 

■  Prior  support  to  government  achieving  cost-effective  cybersecurity  authorizations  to  operate. 

•  Define  measures  and  metrics  clearly  to  evaluate  qualification  of  contractor  cybersecurity  staff. 

•  Degree  to  which  cybersecurity  is  included  in  design  trade  analysis. 

•  Degree  to  which  security  testing  is  integrated  into  software  development. 

■  Degree  to  which  supply  chain  risk  management  ensures  security  and  integrity  of  sourced  components. 

■  Degree  to  which  supply  chain  diversity  is  implemented. 

Key  to  Table:  CDRL=Contract  Data  Requirements  List;  DID=Data  Item  Description;  DoDI  =  DoD  Instruction;  DoDD=DoD  Directive; 
RMF=  Risk  Management  Framework;  SE=Systems  Engineering;  SRD=System  Requirements  Document;  SSE=Systems  Security  Engi¬ 
neering;  T&E=Test  and  Evaluation;  TRD=Technical  Requirements  Document. 

Sources:  DoD  Program  Manager's  Guidebook  for  Integrating  the  Cybersecurity  Risk  Management  Framework  (RMF)  into  the  System  Acquisi¬ 
tion  Lifecycle ;  the  Guide  for  Integrating  Systems  Engineering  into  DoD  Acquisition  Contracts ;  Suggested  Language  to  Incorporate  System  Secu¬ 
rity  Engineering  for  Trusted  Systems  and  Networks  into  DoD  Requests  for  Proposals ;  and  https://shortcut.dau.mil/ncma/cyber_contracts. 
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base  and  stable  workforce  employment.  Incentives  also  can 
be  either  positive,  negative  or  a  combination  of  both.  They 
should  be  applied  selectively  to  motivate  contractor  efforts 
that  otherwise  might  not  be  emphasized  and  to  discourage 
suboptimal  performance. 

When  it  comes  to  incentives,  we  must  always  strive  to  have 
a  better  understanding  of  what  incentives  do  and  make  sure 
that  we're  incentivizing  the  correct  behavior.  Early  market 
research  is  the  key  to  doing  this  successfully.  For  one  thing, 
in  using  multiple  incentive  arrangements,  we  need  to  en¬ 
sure  that  we  always  include  a  cost  incentive  so  that  the 
contractor  doesn't  exceed  contractual  costs  by  chasing  that 
incentive.  We  also  need  to  ensure  that  multiple  incentives 
are  not  driving  suboptimal  performance  in  other  areas— or 
contradicting  one  another. 

The  development  of  an  effective  acquisition  strategy  begins 
with  understanding  the  program's  cybersecurity  requirements 
and  making  a  thorough  evaluation  of  risk.  Contract  incentives 
must  properly  motivate  the  contractor.  Hence,  we  must  un¬ 
derstand  factors  that  are  most  important  to  the  contractor. 

Contract  Type  Challenges  for  Cybersecurity 

Factors  to  consider  when  selecting  a  contract  type  include  (1) 
performance  risk  and  uncertainty,  (2)  urgency,  complexity  and 


stability  of  the  requirement,  (3)  competition  and  (4)  technol¬ 
ogy  maturity.  A  challenge  for  cybersecurity  is  the  availability 
of  historical  cost  and  pricing  data  as  we  build  cybersecurity 
into  the  design  of  systems  as  opposed  to  using  a  previous  com¬ 
pliance  checklist  approach.  The  ever-increasing  cyber  threat 
drives  up  uncertainty  as  new  vulnerabilities  are  discovered 
daily.  As  we  tackle  this  threat,  the  contract  type  needs  to  give 
us  the  flexibility  to  make  adjustments  as  we  learn  what  is  fea¬ 
sible  and  affordable. 

Summary 

The  contracting  community  has  a  crucial  role  to  play  in  ensur¬ 
ing  cybersecurity  requirements  are  effectively  included  in  the 
contract.  This  starts  with  gaining  a  complete  understanding 
of  the  program  requirements  so  that  the  solicitation  can  be 
effectively  constructed  to  differentiate  between  competing 
offerors'  proposals  and  determine  their  capability  to  deliver 
cybersecurity.  The  program  management  office  needs  to  ef¬ 
fectively  communicate  requirements  to  industry  partners  so 
they  understand  the  scope  of  those  requirements.  This  cannot 
be  done  effectively  without  early  engagement  on  the  part  of 
the  contracting  officer.  & 
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Instruction,  Direction 
and  Correction 


Improving 

the  Acquisition  Culture 

James  N.  Phillips  Jr.,  DBA,  PMP,  CFCM 


Criticism  of  the  acquisi¬ 
tion  profession  com¬ 
monly  focuses  on  sus¬ 
tained  performance  and 
positive  outcomes.  The 
problem  appears  to  be  that 
our  acquisition  culture  sug¬ 
gests  that  it  is  unnecessary  to 
take  time  to  improve  people  and  pro¬ 
cesses.  The  acquisition  field  is 
replete  with  courses  offered  by 
government  and  industry  on 
foundational  learning  subjects 
that  are  required  by  the  Depart¬ 
ment  of  Defense  under  the  Defense 
Acquisition  Workforce  Improvement  Act 
(DAWIA)  and  by  the  Federal  Acquisition  In¬ 
stitute  for  civilian  agencies.  With  so 
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Figure  1.  Supervisory  Management  Wheel 


many  courses  available,  why  are  there  still  problems  with  ac¬ 
quisition  performance  and  outcomes? 

Perhaps  the  answer  lies  in  a  holistic  understanding  of  how 
managers  and  supervisors  value  learning,  both  foundational 
and  applied.  Presently  most  training  is  linked  to  the  attainment 
of  certifications— i.e.,  DAWIA  Level  III  or  Federal  Acquisition 
Certification  in  Contracting  (FAC-C)  Level  III— rather  than 
building  the  professional's  qualifications.  The  current  thinking 
seems  to  be  built  on  the  assumption  that  acquisition  profes¬ 
sionals  are  to  be  regarded  as  universally  competent  once  they 
are  FAC-C  Level  III  or  DAWIA  Level  III  certified. 

This  belief  is  rooted  in  the  faulty  notion  that  certification  equals 
qualification.  This  tendency  to  focus  on  the  certification  leaves 
out  the  unique  contribution  that  qualification  brings.  So  how 
do  we  bridge  the  gap  between  foundational  learning  and  ap¬ 
plied  learning? 

The  Value  of  Reflection 

A  colleague  once  shared  with  me  a  pertinent  perspective: 

Would  we  allow  a  newly  minted  second  lieutenant  start  to  fly  a 
$100  million  aircraft  without  first  completing  specific  training? 

So  why  would  we  allow  an  inexperienced  contracting  officer  to 
"'fly"  a  $100  million  procurement? 


tion  teams.  In  2015,  I  created  the  Supervisory  Manage¬ 
ment  Wheel  (hereafter  referred  to  simply  as  Wheel)  to 
help  better  understand  the  relationship  between  Outcome 
(Production)  and  Continuous  Improvement  (Reflection). 
I  particularly  wanted  to  study  the  opposite  side  of  the 
coin— Reflection— which  includes  critical  thinking,  prob¬ 
lem-solving,  and  continuous  improvement. 

Outcome  is  based  on  performance,  which  is  linked  to  founda¬ 
tional  and  workflow  learning  and  Continuous  Improvement  is 
found  in  Reflection  and  is  rooted  in  performance  learning.  The 
Wheel  is  a  model  and  offers  plenty  of  overlap  to  its  application, 
so  it  serves  as  a  guide  and  not  as  an  absolute. 

The  Wheel’s  Center 

The  Wheel  begins  with  three  fundamental  actions  that  any 
supervisor  or  manager  must  conduct  when  managing  out¬ 
comes.  These  actions  are  Instruction,  Direction,  and  Correc¬ 
tion.  Instruction  is  where  foundational  learning  and  certifica¬ 
tions  reside.  Instruction  is  the  "how  to"  part  of  performance 
in  a  theoretical  sense.  Direction,  on  the  other  hand,  reflects 
actual  performance  of  work  or  applied  knowledge.  It  is  in  Di¬ 
rection  that  the  supervisor  defines  work-flow  to  standardize 
practice  and  outcomes.  Finally,  in  Correction,  performance 
learning  occurs!  This  is  where  the  system  corrects  and  im¬ 
proves  itself  by  questioning  its  methods  and  practice.  It  is  in 
Correction  that  continuous  improvement  resides. 

The  First  Ring  Segments 

The  first  ring  segments  overlap  the  actions  of  the  center,  for 
instance,  the  fruit  of  Instruction  and  Direction  is  Performance. 
Performance  suggests  foundational  learning,  and  workflow 
learning  to  produce  an  outcome.  Compliance  combines  both 
Direction  and  Correction,  suggesting  workflow  learning  and 
performance  learning;  and  the  Improvement  segment  com¬ 
pletes  the  ring. 

Improvement  combines  the  elements  of  Correction  and  In¬ 
struction,  in  that  order.  Improvement  is  the  fruit  of  Correction 
and  Instruction  updating.  It  is  in  the  Improvement  segment 
that  challenges  to  work  processes  are  brought  to  light  and 
that  there  is  continuous  improvement  analysis.  Consider  the 
following  scenario: 

Julie  is  a  contracting  officer  with  a  DAWIA  Certified  Level 
III  (Instruction)  and  assigned  to  an  important  project  for  her 
agency  (Direction).  After  receiving  her  marching  orders,  Julie 
discovers  that  the  agency's  process,  which  she  was  follow¬ 
ing,  no  longer  makes  sense  in  the  situation.  Julie  reviews  the 
matter  and  then  offers  alternatives  that  will  reflect  certain 
performance  economies  (Correction).  After  discussing  her 
work  with  her  supervisor/manager,  Julie  is  able  to  document 
the  file  and  make  the  changes  in  the  process. 


In  fact,  the  pilot  would  never  fly  the  jet  without  having 
been  qualified  on  a  simulator.  However,  the  expectation 
is  not  the  same  for  contracting  officers  and  their  acquisi- 


Julie  is  required  to  follow  the  prescribed  contracting  processes. 
Julie  also  is  empowered  to  question  the  processes  and  is  en¬ 
couraged  to  offer  alternative  solutions  for  consideration. 
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Production-Reflection  Balance 

Finally,  two  arrows  encircle  Wheel,  and  these  involve  Produc¬ 
tion  and  Reflection.  The  arrows  suggest  the  Wheel  has  a  dy¬ 
namic  characteristic  similar  to  the  ebb  and  flow  of  outcomes; 
it  is  the  consummate  yin  and  yang  relationship.  In  contracting, 
there  are  times  to  produce  and  times  to  reflect,  a  fact  that 
Julie  understood. 

One  often  hears  talk  about  work-life  balance.  It  is  recognized 
that  balancing  work  and  personal  life  is  essential  to  a  healthy 
employee  and  positive  outcomes.  Similarly,  an  understand¬ 
ing  is  needed  that  another  balance  must  be  struck  when  an 
employee  is  at  work.  Supervisors  and  managers  must  allow 
for  time  to  reflect,  which  is  captured  in  the  phases  illustrated 
in  the  Production-Reflection  dynamic  relationship  found  in 
the  Wheel. 

This  balance  is  theoretical.  Unfortunately,  the  scale  often  is 
tilted  toward  the  Production  phase  rather  than  giving  equal 
time  in  the  Reflection  phase  where  continuous  improvement 
lies  and  where  performance  learning  occurs.  Reflection  super¬ 
charges  understanding  and  re-energizes  the  willingness  of  the 
individual  to  exercise  greater  initiative.  In  his  book  Seven  Habits 
of  Highly  Effective  People ,  the  late  Dr.  Stephen  Covey  noted  that 
one  of  the  habits  is  "Sharpening  the  Saw,"  which  is  particularly 
relevant  here.  In  "Sharpening  the  Saw,"  Covey  reminded  the 
reader  of  the  importance  of  down  time  and  preparation  for 
work.  Deliberately  taken  down  time  affords  time  to  reflect, 
recharge,  and  renew  a  person's  view  of  his  or  her  work. 

Discussion 

The  Wheel  is  a  conceptual  model,  an  image  of  ideal  opera¬ 
tional  culture  and  its  interrelationships.  It  lays  out  in  simple 
imagery  key  actions  and  relationships  on  which  supervisors 
and  managers  of  acquisition  personnel  must  focus. 

Acquisition  excellence  does  not  happen  in  vacuum.  Acquisition 
excellence  is  created  when  the  culture  within  the  contracting 
department  and  acquisition  community  takes  time  to  reflect. 
This  means  analyzing  past  performance  by  using  critical  think¬ 
ing  skills  to  formulate  a  plan  for  the  future.  This  may  sound 
familiar  as  it  reflects  the  PDCA  (Plan-Do-Check-Act)  and  is 
captured  in  the  Sprints  used  in  Agile  Software  Development. 

The  Reflection  phase  is  where  continuous  improvement  oc¬ 
curs  and  is  tested  and  recommended  for  implementation.  Fol¬ 
lowing  Lean  principles,  there  must  be  time  set  aside  to  reflect 
so  that  learning  can  occur.  For  the  contracting  supervisor  and 
manager,  this  is  where  cataloging  Corrections  and  Compli¬ 
ance  concerns  can  be  turned  into  learning  moments  where 
Improvement  may  occur.  Consider  the  following  scenario: 

Joseph  is  a  contracting  officer  with  a  DAWIA-certified  Level 
III  and  was  recently  reassigned  from  a  station  level  contract¬ 
ing  shop  to  an  important  contract  in  the  F-35  program  office. 
While  Joseph  was  certified,  he  was  not  fully  qualified  for  his 
new  role  as  he  had  no  program  environment  background. 


In  his  new  assignment,  the  program  manager  (PM)  made 
certain  that  he  had  instruction  on  the  basic  understanding 
of  the  program  and  its  contracts  (Compliance  segment).  The 
PM  then  ensured  that  Joseph  had  applied  his  new  knowl¬ 
edge  of  the  program  on  challenging,  but  familiar,  contracts 
so  that  Joseph  could  understand  how  business  was  done  at 
the  program  office.  Finally,  Joseph  was  given  more  complex 
contracts  and  had  greater  responsibility  for  the  outcome. 
Joseph  was  learning  and  applying  the  lessons  learned  to 
future  contracts  (Improvement  segment),  and  sharing  the 
lessons  learned  with  his  colleagues  and  helping  them  along 
(Performance  segment). 

Ideas  on  Reflection 

The  Reflection  phase  includes  lessons  learned,  which  is  a 
retrospective  understanding  of  prior  actions,  and  simula¬ 
tion,  which  is  a  prospective  view  of  what  is  to  come.  Both 
lessons  learned  and  simulation  will  aid  in  developing  greater 
contracting  situational  awareness  when  facing  new  pro¬ 
curement  actions.  Supervisors  and  managers  must  use  both 
to  prepare  acquisition  professionals  for  the  challenge  of  new 
procurement,  and  this  can  best  be  performed  during  the 
Reflection  phase. 

Final  Thoughts 

The  Wheel  should  be  a  staple  item  for  all  supervisors  and 
managers,  as  well  as  mentors  and  coaches.  It  visually  conveys 
an  understanding  of  the  relationship  of  the  Reflection  phase  to 
the  whole  of  employee  development.  The  design  of  the  Wheel 
illustrates  the  many  overlapping  elements.  The  Wheel  when 
viewed  from  the  center  presents  interconnected  building 
blocks  necessary  for  good  outcomes.  When  viewed  from  the 
outside  in,  the  Wheel  presents  a  tapestry  of  interconnected 
relationships  that  are  crucial  for  sustaining  outcomes. 

Supervisors  and  managers  create  meaningful  and  relevant 
learning  opportunities  when  the  learning  is  planned.  I  found 
the  Wheel  most  useful  in  its  elevation  of  the  value  of  the 
Reflection  phase  to  the  same  level  as  the  Production  phase. 
The  Production  phase  is  designed  to  produce,  not  change 
on  the  fly.  Production  is  the  fruit  of  foundational  learning, 
instructions,  standard  work,  leading  to  a  positive  outcome; 
whereas  Reflection  focuses,  improving  the  outcomes  either 
by  better  compliance  or  improved  processes.  As  referred 
to  earlier,  in  the  "down  time"  of  Reflection  critical  thinking 
is  performed  and  alternatives  are  discovered.  This  is  the 
home  of  continuous  improvement  and  also  where  simula¬ 
tion  lives.  The  second  lieutenant  mentioned  above  definitely 
would  be  put  in  a  simulator  before  actually  flying  the  $100 
million  aircraft.  We  should  make  certain  that  the  same  de¬ 
liberate,  planned  learning  occurs  before  a  contracting  officer 
jumps  into  a  $100  million  procurement.  In  final  reflection,  the 
Wheel  provides  acquisition  professionals  a  balanced  under¬ 
standing  with  which  to  work  (Production)  and  to  prepare  for 
work  (Reflection). 


The  author  can  be  contacted  at  james.n.phillipsjr@gmail.com. 
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Successful  Department  of  Defense 
(DoD)  acquisitions  are  the  product  of 
comprehensive,  structured  and  ongo¬ 
ing  due  diligence  strategies,  custom  tai¬ 
lored  for  each  phase  of  each  program. 
Program  managers  (PMs)  are  only  half  right  to 
believe  that  due  diligence  is  reactive  and  starts 
with  the  proposal. 

Due  diligence  must  be  proactive  as  well  and  start  with  the  Needs 
Assessment.  PMs  must  be  equally  industrious  when  initially 
identifying  needs  and  developing  the  requests  for  proposals 
(RFPs)— and  then  throughout  the  remaining  acquisition  process. 


" Quality  products  that 
satisfy  user  needs  with 
measurable  improvements 
to  mission  capability  and 
operation  and  support,  in  a 
timely  manner  at  a  fair  and 
reasonable  price." 

—Department  of  Defense 


Razzetti  is  a  retired  US.  Novy  captain,  management  consultant,  auditor,  and 
military  analyst  He  is  the  author  of  five  management  books,  including  a  book 
for  the  military:  " Hardening  by  Auditing— A  Handbook  for  Measurably  and 
Immediately  Improving  the  Security  Management  of  Any  Organization"  and 
has  served  on  the  advisory  boards  of  two  business  schools. 
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Figure  1.  Due  Diligence  in  Every  Step 

Due  Diligence 

(Concept  of  Operations,  Program  reviews,  audits,  site  visits,  milestone  achievement,  test  plans, 
expenditure  tracking,  metrics/measures  of  effectiveness,  user  feedback) 
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Source:  Adapted  by  the  author  from  DoD  Directive  5000.1. 


Figure  1  outlines  a  generic  DoD  acquisition  process,  suggesting 
to  PMs  that  there  is  both  a  need  and  an  opportunity  for  due 
diligence  at  every  program  phase. 

Management  books  define  due  diligence  as  "investigation  by 
or  on  behalf  of  an  intended  buyer  of  a  product  or  service  to 
check  that  the  seller  has  the  desired  assets,  turnover,  profits, 
market  share  positions,  technology,  customer  franchise,  pat¬ 
ents  and  brand  rights,  contracts  and  other  attributes  required 
by  the  buyer  or  claimed  by  the  seller." 

In  the  private  sector,  designated  due  diligence  personnel  (e.g., 
a  team  of  financial,  technical  and/or  legal  experts)  review 
and  analyze  all  operative  documents  submitted  by  potential 
contract  awardees.  Moreover,  growing  numbers  of  business 
enterprises  are  pursuing  additional  legal  protection  for  them¬ 
selves  in  order  to  shield  themselves  from  harm  if  their  due 
diligence  efforts  fail  to  uncover  serious  problems  with  mergers 
or  purchase  transactions. 

For  our  purposes,  due  diligence  in  acquisitions  means  making 
certain  that  all  the  facts  regarding  an  organization  are  available 
and  have  been  measurably  verified.  More  on  this  later. 

Effective  due  diligence  processes  include  Environmental  due 
diligence,  like  environmental  site  assessments  to  avoid  liabil¬ 
ity  underthe  Comprehensive  Environmental  Response,  Com¬ 
pensation,  and  Liability  Act  (CERCLA),  commonly  referred  to 
as  the  "Superfund  law."  Manufacturing  due  diligence  involves 
a  number  of  concepts  regarding  either  the  performance  of 
source  inspections  or  surveillances,  such  as  quality  system 
audits.  Due  diligence  in  contractor  quality  is  the  effort  made 
by  safety,  quality  and  environmental  professionals  to  validate 
conformance  provided  by  sellers  to  purchasers.  Investigative 
due  diligence  involves  a  general  obligation  to  identify  true 


root  causes  for  noncompliance  on  a  standard  or  contract 
requirement. 

Failure  to  exert  due  diligence  may  (and  perhaps  should)  be 
considered  negligence. 

In  performance,  due  diligence  audits  are  very  similar  to  any 
other  audits.  I  advise  clients  that  it  may  be  less  complicated 
just  to  think  of  due  diligence  as  a  part  of  their  day-to-day  man¬ 
agement  strategy,  like  any  other  internal  control. 

Identifying  the  Requirement — 

First  Things  First 

DoD  cannot  expect  contractors  to  create  spot-on  products 
or  actionable  services  unless  it  is  precise  in  the  development 
and  specification  of  its  requirements.  The  Needs  Assessment 
and  the  research  it  both  entails  and  generates  impose  ongo¬ 
ing  due  diligence  demands  and  expectations  on  DoD.  Only 
the  most  scrupulous  developmental  processes  will  do  for  the 
"Buy  or  Cancel  Decision,"  and  a  flawed  Statement  of  Work  will 
inevitably  produce  a  flawed  product  or  service.  See  Figure  2. 

All  this  before  the  contractor  even  gets  a  peek  at  the  RFP. 

The  RFP 

A  great  deal  has  been  written  about  the  RFP  process,  primar¬ 
ily  regarding  the  U.S.  Government  and  its  formal  acquisition 
programs.  You  can  find  just  as  much  written  about  how  con¬ 
tractors  answer  RFPs  with  (seemingly)  credible  and  executable 
proposals  and  their  plans  to  achieve  the  success  expected  fol¬ 
lowing  contract  award.  We  could  discuss  that  all  day,  but  we 
will  stay  with  what  you  need  to  do  to  impress  upon  contractors 
that  to  bid  for  DoD  business  is  to  perform  in  an  atmosphere 
of  mutual  honesty,  mutual  understanding,  and  mutual  ben¬ 
efit.  DoD  must  impress  on  contractors  its  seriousness  and 
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Figure  2.  Due  Diligence  Throughout 
the  Acquisition 
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Source  for  Figures  2  and  3:  The  author. 


commitment— and  that,  if  the  product  does  not  achieve  the 
goals  for  which  it  was  built,  a  heavy  cost  could  be  exacted  in 
mission  failures  and  losses  of  life. 

The  RFP's  size  and  complexity  are  functions  of  the  work  re¬ 
quested.  However,  regardless  of  the  physical  size,  the  RFP 
must  include  many  fundamentals,  each  well-researched  and 
unambiguous.  Technical  specification  of  the  required  product 
or  service  desired  should  be  as  precise  as  possible.  Include 
an  abbreviated  management  plan,  again  containing  material 


previously  developed  (e.g.,  objectives),  plus  organizational  re¬ 
sponsibilities,  interfaces,  reporting  requirements,  regulatory 
requirements  and  schedules. 

The  RFP  is  the  sum  of  all  the  research,  analyses,  and  intelli¬ 
gence  collection  that  you  have  done.  It  must  be  scrupulously 
performed  and  just  as  scrupulously  reflected  in  the  RFP,  if  the 
acquisition  is  expected  to  obtain  the  product  or  service  that 
the  troops  need. 

How  well  the  contractor  understands  and  is  willing  to  comply 
is  the  subject  of  the  next  section. 

The  Proposal 

An  ethical  contractor,  like  an  ethical  management  consultant, 
should  never  bid  on  a  job  he  or  she  cannot  do  well. 

Proposals  must  do  more  than  answer  the  mail— they  must 
answer  the  need. 

Upon  receiving  the  RFP,  it  is  hoped  that  the  contractor  will 
analyze  it  thoroughly  to  determine  whether  there  is  an  ad¬ 
vantage  to  be  gained  in  responding  with  an  offer  of  work.  The 
contractor's  proposal  should  address  every  point  of  the  RFP, 
in  accordance  with  the  stated  provisions. 

Some  companies  have  business  development  personnel 
ready  to  respond  to  any  RFP,  either  by  themselves  or  with 
a  staff  of  nameless,  faceless,  "cut  and  paste"  commandos. 
Resulting  proposals  often  are  mosaics  of  favorite  blurbs  from 
previous  proposals.  The  objective:  Get  the  contract  first,  and 
then  worry  about  how  to  perform  the  work.  Be  afraid— be 
very  afraid! 

Proposals  require  the  greatest  possible  due  diligence  from 
both  the  contractor  and  the  DoD  PM.  Review  must  go  far  be¬ 
yond  block-checking  and  page-counting  by  cubicle-dwellers. 
Proposals  must  be  more  than  just  correct  and  comprehensive. 
They  must  be  forthright,  straightforward  and  free  of  deception, 
credible  beyond  question,  and  scrupulously  reflect  the  state 
of  the  contractor's  organization  and  management  approach. 
The  Proposal  is  "cradle  to  grave";  it  should  describe  the  entire 
life  cycle  of  the  product  or  service. 

The  Statement  of  Work  must  be  as  it  was  written,  but  now  with 
the  contractor's  description  of  how  it  will  perform  your  tasks. 
This  is  critical.  The  proposal  must  be  responsive  to  the  DoD's 
needs  as  specified  in  the  RFP,  complete  with  performance  re¬ 
quirements  and  measures  of  effectiveness. 

The  Management  Approach— again  restating  yours  but  with 
the  contractor's  execution  plan,  specifies  that  the  design  pro¬ 
cess  (if  appropriate)  is  adequately  defined  and  incorporates 
appropriate  technologies,  such  as  computer-aided  design; 
databases  are  comprehensive  and  test  and  evaluation  pro¬ 
cedures  are  established  or  confirmed,  and  life-cycle  require¬ 
ments  are  defined. 
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A  comprehensive  proposal  also  should  include  (in  some  for¬ 
mat  or  another): 

■  A  quantifiable  summary  of  the  organization's  perfor¬ 
mance  track  record  with  similar  projects 

■  Adherence  to  statutory  and  regulatory  requirements 

■  A  vision  of  the  outcome 

■  Products  and  services  and  their  end  uses 

■  Comparison  of  the  organization's  products  and  services 
with  those  of  potential  adversaries 

■  Warranties,  guarantees,  and  follow-on  service 

■  Synergies  and  innovations 

■  A  formidable  understanding  of  the  threat  necessitating 
the  product  or  service  and  whether  the  products  or  ser¬ 
vices  address  the  threat  for  which  they  were  developed 

■  Post-delivery  service  support  (hotline,  maintenance,  com¬ 
plaints,  upgrades,  acquisition  from  stateside  sources  not 
deployed  forward,  etc.) 

There  should  also  be  evidence  (often  separately  provided)  of 
contractor  soundness  and  the  assumptions  underlying  that 
soundness  (i.e.,  will  the  contractor  go  under  if  it  does  not  win 
the  contract?). 

A  contractor,  in  order  to  get  the  job,  may  underbid  (i.e.,  "low 
ball")  the  competition,  often  expecting  to  recoup  lost  money 
in  amendments,  modifications  and  extensions.  They  often 
succeed,  but  they  just  as  often  wind  up  working  nights  and 
weekends  "for  free"  because  the  money  to  pay  for  all  those 
deliverables  simply  isn't  there.  Another  unacceptable  reac¬ 
tion  is  for  the  contractor  to  assign  the  work  to  less-qualified 
personnel  because  the  cost  is  less  than  that  of  the  original 
personnel  assigned.  When  this  is  planned  at  the  outset,  it's 
often  called  "bait  and  switch"  and  is  unethical.  In  any  event, 
this  risks  delivering  a  low-quality  product  or  service,  not  what 
is  being  paid  for— and,  even  worse,  not  what  the  troops  need. 

When  the  contractor  complains  to  his  or  her  congressman, 
you  really  need  to  have  your  act  together. 

Review  the  proposal  carefully,  to  ensure  that  you  will  be  getting 
exactly  what  you  asked  for  within  the  time,  funding,  and  quality 
constraints  you  stated— before  you  sign  on  the  dotted  line. 

Execution 

From  the  first  moment  of  the  acquisition  process,  PMs  oper¬ 
ate  in  a  "triple  threat"  environment,  as  gloomily  described  in 
Figure  3.  Performance  is  critical,  and  the  reader  is  reminded 
that  there  should  be  no  doubt  in  anyone's  mind  about  what  the 
product  is  supposed  to  do.  That's  why  the  Concept  of  Opera¬ 
tions  and  the  Statement  of  Work  must  be  scrupulously  devel¬ 
oped,  understood  and  followed.  Time  and  cost  often  have  an 
inverse  relationship  with  performance.  That  is,  the  contractor 
often  wants  more  time  and  funding,  in  return  for  lower  product 
expectations.  Again,  program  success  in  the  triple  threat  envi¬ 
ronment  requires  due  diligence  in  internal  controls— constant 
and  unwavering. 


Figure  3.  The  Triple  Threat  Environment 


TIME 


And  all  credible  organizations,  military  and  civilian,  require 
comprehensive  and  meaningful  internal  controls.  A  due  dili¬ 
gence  audit  of  these  controls  should  judge  not  only  the  prod¬ 
ucts  of  the  controls  (e.g.,  records  and  reports),  but  also  the 
sufficiency  and  comprehensiveness  of  the  controls  themselves 
and  the  level  of  importance  and  relevance  attached  to  them  by 
the  contractor.  Private  sector  organizations  that  let  their  inter¬ 
nal  control  processes  slide,  or  do  not  take  action  on  problems 
surfaced  by  the  controls,  deservedly  lose  their  credibility— 
maybe  even  their  existence. 

Acquisition  managers  may  need  outside  help  from  addi¬ 
tional  personnel  with  specialized  experience,  expertise  or 
certification. 

Summary 

Many  organizations  in  both  the  public  and  private  sectors  un¬ 
dertake  the  due  diligence  process  with  insufficient  vigor.  In 
some  cases,  the  prevailing  culture  suffers  from  malaise  and 
views  due  diligence  as  a  perfunctory  exercise  to  be  checked 
off  quickly.  In  other  instances,  the  outcome  of  the  due  diligence 
process  may  be  tainted  (either  consciously  or  unconsciously) 
by  stakeholders  who  stand  to  benefit  personally  or  profession¬ 
ally  from  contract  awards. 

DoD  must  guard  against  such  casual  or  flawed  attitudes  from 
impacting  its  programs.  A  robust  and  actionable  DoD  due  dili¬ 
gence  strategy  can  prevent  costly  failures— measured  both  in 
lives  and  funds.  Moreover,  failures  and/or  shortcomings  in  one 
mission  area  (e.g.,  intelligence  collection)  also  can  adversely 
impact  related  missions  (e.g.,  power  projection).  Those  same 
failures  will  have  profound  consequences  not  only  on  our  se¬ 
curity,  but  on  our  national  posture  and  international  reputation. 

I  close  with  my  favorite  quote  from  19th-century  jurist  Edmund 
Burke:  "The  only  thing  necessary  for  evil  to  triumph  is  for  good 
men  to  do  nothing." 

The  author  can  be  reached  at  generazz@aol.com. 
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Schultz  is  a  professor  of  Program  Management  at  the  Defense  Acquisition  University's  Fort 
Belvoir,  Virginia ,  campus. 


Brian  Schultz 


"Strategy  101  is 
about  choices:  You 
can't  be  all  things 
to  all  people/' 

—Michael  Porter,  noted 
academic  expert  on 
strategic  thinking 


The  guidance  from  Department  of  Defense 
(DoD)  leadership  is  very  clear  when  it  comes 
to  developing  acquisition  strategies.  Every 
program  should  consider  and  propose  tailor¬ 
ing  of  information,  work  efforts,  and  decision 
reviews  if  this  tailoring  will  result  in  a  more  cost-ef¬ 
fective  approach.  The  words  tailor  and  tailoring  ap¬ 
pear  48  times  in  the  Aug.  10,  2017,  DoD  Instruction 
5000.02.  However,  as  is  the  case  with  most  com¬ 
plex  acquisition  tasks,  no  single  cookbook  solution 
will  work  for  every  program. 
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Figure  1.  Integrating  Acquisition  Strategy  Elements 
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Source:  WSM  014  Integration  Topic  (Defense  Acquisition  University  workshop) 


As  the  Michael  Porter  strategy  quote  suggests,  an  important 
part  of  any  strategy  is  making  good  choices,  including  what 
not  to  do  or  pursue.  In  the  context  of  a  corporate  business 
strategy,  a  company  typically  will  determine  what  it  is  good 
at  and  then  use  that  core  competency  as  a  competitive  ad¬ 
vantage  to  grow  its  business  and  increase  shareholder  value. 
Companies  realize  they  cannot  be  everything  to  everybody,  so 
they  play  to  their  strengths  and  choose  to  avoid  opportunities 
where  they  are  weak.  This  mind-set  of  playing  to  strengths, 
choosing  opportunities  carefully,  and  determining  what  not  to 
pursue  also  applies  to  DoD  acquisition.  DoD  program  manag¬ 
ers  (PMs)  should  consider  similar  factors  when  developing  an 
acquisition  strategy. 

When  I  participate  in  workshops,  courses,  and  other  training 
events  with  DoD  acquisition  workforce  staff,  I  often  am  asked 
how  we  should  go  about  streamlining  and  tailoring  the  pro¬ 
gram's  acquisition  strategy.  This  topic  is  the  central  theme  of 
our  Acquisition  Strategy  Development  Workshop  (also  known 
as  WSM  014  in  the  Defense  Acquisition  University  i-catalog). 
The  following  discussion  captures  some  of  the  many  of  the 
points  I  try  to  convey  based  on  my  PM  experiences.  While  the 
final  strategy  will  reflect  the  key  tailoring  decisions,  there  are 
some  fundamental  building  blocks  that  should  be  considered. 

Start  With  Program  Priorities.  It  may  seem  obvious,  but 
PMs  need  to  determine  the  priorities  that  will  drive  their 
strategy.  These  priorities  are  not  to  be  confused  with  under¬ 


standing  and  analyzing  user  requirements  and  the  resultant 
trade-space,  some  of  which  will  be  executed  after  contract 
award  in  a  development  effort.  The  starting  point  should 
be  the  upfront  analysis  to  assess  the  relative  importance  of 
cost  versus  schedule  versus  performance.  For  example,  there 
are  trade-offs  associated  with  prioritizing  schedule  over  cost 
and  performance  and  this  determination  will  drive  the  overall 
strategy  and  tailoring  decisions. 

On  the  other  hand,  a  program  may  be  more  willing  to  accom¬ 
modate  some  schedule  slippage  and  cost  growth  to  achieve 
greater  technical  performance.  Understanding  this  relative 
importance  can  drive  other  strategy  considerations,  includ¬ 
ing  scope,  constraints  and  even  contract  type  and  structure. 
This  analysis  should  involve  acquisition  leadership,  user  and 
test  communities  to  ensure  a  common  understanding  and 
alignment  of  stakeholder  expectations.  The  government  is 
required  in  the  request  for  proposal  (RFP)  to  state  the  relative 
of  importance  of  cost  versus  no-cost  factors,  but  this  should  be 
determined  well  in  advance  of  developing  the  RFP.  It  is  useless 
to  tailor  a  strategy  without  thinking  through  the  cost,  schedule 
and  performance  priorities  upfront. 

A  few  years  back,  my  team  managed  an  urgent  surveillance 
radar  program  that  was  to  be  deployed  to  support  emergent 
combat  operations.  Based  on  the  importance  of  schedule,  we 
structured  the  program  to  accelerate  tasks  and  conduct  con¬ 
currency  in  site  design  and  radar  production.  We  recognized 
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that  some  aspects  of  the  program,  including  quality  and  cost, 
would  not  be  optimized  based  on  rushing  everything  from  con¬ 
tract  award  to  production  sequencing  and  site  deployment. 

I  have  observed  programs  that  were  essentially  procuring 
commercial-off-the-shelf  (COTS)  equipment  but  had  program 
constraints  that  resulted  in  a  longer-than-expected  fielding 
cycle.  An  example  was  a  program  with  a  requirement  to  con¬ 
duct  a  significant  operational  test  on  the  COTS  equipment 
before  it  was  approved  for  full-rate  production  and  fielding. 
This  testing  would  add  at  least  18  months  to  the  schedule 
for  initial  operating  capability.  Rather  than  wait  for  a  limited 
procurement  of  the  COTS  items  as  test  assets,  followed  by  a 
gatekeeper  event  like  Initial  Operational  Test  and  Evaluation, 
this  program  could  consider  an  alternate  path.  Since  the  item 
is  COTS  and  is  used  by  others,  its  effectiveness  and  suitabil¬ 
ity  already  have  been  demonstrated  for  those  users  who  are 
operating  it.  Assessing  the  test  and  operational  data  available 
from  these  existing  users  could  eliminate  the  need  for  another 
lengthy  test  cycle.  Using  new  COTS  equipment  in  operational 
demonstrations  and  exercises  also  could  supplement  the  ex¬ 
isting  test  data  to  help  assess  whether  the  COTS  equipment 
should  be  acquired. 

Perform  Prerequisite  Tasks  and 
Employ  Critical  Thinking 

In  order  to  make  informed  tailoring  decisions,  PMs  should  ex¬ 
amine  evidence  that  supports  the  proposed  tailoring.  In  the 
context  of  the  acquisition  strategy,  tasks  such  as  market  re¬ 
search,  identification  of  framing  assumptions,  and  risk  and  op¬ 
portunity  management  are  good  starting  points  to  provide  that 
evidence.  PMs  and  their  teams  may  have  some  preconceived 
notions  about  the  best  alternative  for  the  strategy  and  some 
tasks  that  could  be  tailored.  While  these  notions  may  be  based 
on  experience  and  good  judgment,  PMs  should  keep  an  open 
mind  as  to  other  alternatives  and  new  possibilities.  The  pace 
of  technological  change  is  so  rapid  that  previously  unassessed 
new  approaches  now  may  be  relevant  for  consideration. 

As  an  example,  consider  software  reliant  programs.  The  tradi¬ 
tional  waterfall  method  has  been  used  in  DoD  (and  industry) 
for  decades,  and  has  wide  acceptance.  Based  on  recurring 
information  technology  acquisition  problems,  the  traditional 
waterfall  method  now  is  being  challenged  by  new  methods 
such  as  Agile  and  Agile  DevOps.  These  innovative  software 
development  methods  are  based  on  the  Agile  Manifesto  and 
may  require  policy  or  regulatory  waivers,  depending  on  how 
the  method  is  implemented.  The  acquisition  strategy  is  an  ap¬ 
propriate  vehicle  to  obtain  these  waivers  but  should  be  based 
on  a  sound  business  case. 

Many  commercial  companies  have  employed  cloud  comput¬ 
ing  technology,  automated  test  and  development  tools,  and 
new  methods  to  rapidly  develop  and  deliver  software  to  their 
users  and  customers.  Likewise,  many  DoD  programs  now 
either  are  migrating  their  existing  processes  or  starting  out 
with  this  new  method. 


Critical  thinking  is  necessary  since  it  enables  a  disciplined,  ra¬ 
tional  and  structured  approach  to  help  design  the  way  forward. 
There  are  many  critical  thinking  tools  available  and  we  ad¬ 
dress  one  in  detail  during  the  WSM  014  workshop.  The  team 
developing  the  strategy  should  ensure  that  all  the  key  players 
involved  in  developing  the  strategy  have  the  appropriate  train¬ 
ing  in  the  particular  critical  thinking  approach  employed.  This 
will  help  ensure  a  clear  focus  of  the  thinking  process. 

Another  suggested  technique  is  the  use  of  powerful  ques¬ 
tions  related  to  the  strategy  from  reviewers  independent  of 
the  program  team.  These  questions  can  create  new  aware¬ 
ness  and  challenge  the  status  quo,  generating  additional  ideas 
that  can  help  design  an  effective  strategy.  The  questions  help 
us  to  think  about  possibilities  to  overcome  any  obstacle  that 
will  negatively  affect  cost,  schedule  and  performance  expec¬ 
tations.  The  WSM  014  workshop  conducts  topic  exercises 
around  questions  for  the  team  to  consider  and  they  are  tailored 
to  fit  the  program  circumstances. 

Several  years  ago,  my  team  used  brainstorming  and  critical 
thinking  questions  to  help  identify  a  way  ahead  to  renegotiate 
a  contract  that  was  tainted  by  fraud  involving  a  senior  DoD 
acquisition  official.  What  initially  seemed  an  impossible  task 
was  unexpectedly  executed  with  very  little  difficulty.  We  even 
reduced  the  contract  price  by  more  than  $12  million! 

Consider  the  Industry  Perspective 

Since  the  targeted  customer  of  the  acquisition  strategy  (and 
subsequent  RFP)  is  industry,  looking  at  the  strategy  through 
their  lens  is  vital  to  future  success.  DoD  teams  that  have  little 
or  no  industry  experience  should  seek  assistance  with  this 
task.  Erroneous  assumptions  about  industry  contractors,  in¬ 
cluding  their  commitment,  incentive  to  control  costs,  and  ca¬ 
pability  to  provide  the  product  or  service  within  the  contract 
terms,  can  lead  to  disaster.  I  suggest  to  the  teams  that  solicit¬ 
ing  comments  on  a  draft  RFP  is  only  a  small  part  of  this  effort. 

The  following  are  a  few  examples  of  pre-RFP  information  that 
should  be  considered  and  can  often  be  obtained  through  ap¬ 
propriate  research  efforts: 

Financial  situation  and  motivations.  Not  every  company  is 
motivated  to  control  costs  with  a  cost  or  incentive  type  con¬ 
tract.  I  worked  in  a  business  unit  at  a  commercial  company 
that  made  great  profit  margins  on  most  of  our  work.  I  often 
was  more  concerned  about  growing  my  sales  numbers  at  the 
expense  of  margin  on  many  DoD  contracts.  Teams  also  should 
consider  a  company's  ability  to  absorb  losses  on  a  risky  fixed 
price  contract. 

Strategic  alignment.  Companies  tend  to  focus  on  opportuni¬ 
ties  that  are  closely  aligned  with  their  corporate  strategy.  PMs 
and  their  teams  should  understand  where  the  new  program 
fits  within  company  priorities.  In  some  cases,  industry  will 
invest  to  not  only  win  the  contract  but  also  to  enhance  the 
product  or  service. 
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https://www.dau.miI/tools/t/ 

Request-a-New-Tool 


Take  a  quick  two-question  survey 
on  the  topics,  tools, 
and  resources  you  need. 


Leverage.  Well  define  leverage  as  the  relative  bargaining 
power  of  industry  to  DoD  and  to  a  company's  competitors. 
Knowledge  of  the  mission  area  and  corporate  market  share 
are  good  starting  points  to  consider.  Leverage  can  also  apply 
based  on  time  pressures  and  ability  to  meet  financial  goals. 
When  I  worked  in  industry,  I  could  offer  substantial  discounts 
at  the  end  of  the  quarter  when  I  was  trying  to  hit  my  new 
booking  goal. 

Suppliers  and  outsourced  content.  Given  the  ramp-up  in 
outsourcing  across  multiple  business  domains,  DoD  teams 
need  to  understand  both  the  potential  prime  and  key  suppliers. 
Many  programs  have  seen  cost  and  schedule  issues  associ¬ 
ated  with  subcontractors  and  were  surprised  at  the  extent  of 
the  prime's  outsourcing.  Several  strategy  considerations  will 
be  affected  if  significant  outsourcing  is  expected.  Multiple 
tools  exist  that  can  assist  teams,  but  the  supplier  manage¬ 
ment  risk  mitigations,  to  be  effective,  must  be  planned  early 
and  be  included  in  the  RFP  and  contract. 

The  imperative  to  better  understand  industry  perspectives 
is  gaining  momentum.  In  addition  to  the  DAU  course  ACQ 
315,  Understanding  Industry,  initiatives  such  as  Reverse  In¬ 
dustry  Days  are  becoming  popular.  A  Reverse  Industry  Day  is 
intended  to  provide  an  opportunity  for  government  employees 
to  listen  to  senior  industry  speakers  discuss  industry's  view  of 
market  research,  bid  process,  evaluation  criteria,  incentives, 
bid  pricing,  and  similar  topics. 

Iterate  and  Integrate  the  Strategy 

The  acquisition  strategy  typically  involves  a  very  broad  and 
far  looking  approach.  Bringing  everything  together  in  an  in¬ 
tegrated  manner  is  one  of  the  bigger  challenges.  The  flow  of 
content  development  depicted  in  Figure  1  represents  one  way 
to  develop  this  integration.  The  flow  begins  with  requirements 
and  then  begins  with  the  technical  strategy  that  often  drives 
other,  following  strategy  elements,  including  tailoring  of  these 
elements  to  support  the  technical  approach.  It  is  similar  to  the 
systems  engineering  process  in  that  this  should  be  an  iterative 
and  recursive  effort,  repeating  steps  and  at  different  levels  in 
order  to  optimize  the  strategy. 

Final  Thoughts 

Developing  a  tailored  and  streamlined  acquisition  strategy 
often  is  a  difficult  task.  While  many  tools  and  template  are 
available  to  assist  in  the  development,  there  is  no  one-size- 
fits-all  approach.  The  upfront  planning  and  data  gathering  are 
critical  and  should  be  aimed  at  determining  parameter  impor¬ 
tance,  conducting  prerequisite  planning  tasks,  using  critical 
thinking,  and  considering  the  industry  perspective. 

There  are  not  a  great  deal  of  data  and  research  on  tailor¬ 
ing  approaches,  lessons  learned,  and  benefits  realized.  I  am 
very  interested  to  discuss  and  review  your  thoughts,  lessons 
learned,  and  experiences  on  this  subject.  Thanks  in  advance 
for  your  consideration.  & 

The  author  can  be  contacted  at  brian.schultz@dau.mil. 
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MISSILE  DEFENSE  AGENCY 


How  Commercial 
Space  Spurred 
DoD  Innovation 

Michael  Schlacter 


nyone  who  has  spent  an  appreciable 
amount  of  time  in  Department  of  De¬ 
fense  (DoD)  procurement  knows  that 
the  subject  of  acquisition  reform  sur¬ 
faces  from  time  to  time.  Common  cri¬ 
tiques  are  that  the  process  is  slow  and  unrespon¬ 
sive  to  the  demand  signal,  or  that  the  capabilities 
are  expensive  and  unaffordable.  The  corporate  pro¬ 
cess  responds  to  these  critiques  and  implements 
changes  of  varying  scales  with  the  hope  that,  this 
time,  the  changes  will  be  long  lasting. 


No  mission  area  seems  to  be  immune  from  this  feedback,  and  space  in 
particular  has  been  under  the  microscope  in  recent  years.  The  Govern¬ 
ment  Accountability  Office  (GAO)  has  reviewed  major,  non-launch  space 
programs  more  than  a  dozen  times  since  2010,  comparing  them  by  their 
relative  cost  overruns  and  schedule  slips.  Space  was  also  featured  in  the 
Fiscal  Year  (FY)  2017  National  Defense  Authorization  Act,  Section  1616,  in 
which  legislators  asked  the  DoD  to  look  at  the  management  and  organiza¬ 
tion  of  national  security  space  activities. 

On  the  other  end  of  the  spectrum  is  the  coverage  of  rapid  capability  orga¬ 
nizations  such  as  the  U.  S.  Air  Force's  Operationally  Responsive  Space  and 
Rapid  Capability  Office  organizations.  By  highlighting  these  organizations, 
writers  seem  to  say  there  are  places  where  development  can  occur  quickly 
and  costs  can  be  contained.  Are  there  other  places  where  this  can  happen? 
Yes,  it's  commercial  space. 

Commercial  space  has  transformed  how  the  DoD  looks  at  the  overall  space 
industry.  In  the  past,  the  DoD  relied  mostly  on  large  satellite  programs 
with  military-specific  hardware  and  software.  Today  large  satellites,  small 
satellites,  hosted  payloads  and  managed  services  all  have  the  ability  to 
fill  military  niches  for  which  they  are  uniquely  suited,  and  the  explosion  in 
commercial  space  lift  is  nothing  if  not  exciting. 

Can  commercial  space  encourage  the  DoD  to  change?  Absolutely— and 
not  just  on  the  surface  like  the  DoD  using  a  commercial  products  and 
services  contract  strategy.  Instead,  the  change  can  occur  at  the  funda¬ 
mental  management  level  by  suggesting  a  new,  innovative  way  to  lead 
DoD  space  projects. 

One  DoD  space  program  matched  the  speed  of  its  commercial  partners 
and  gained  benefits  when  it  achieved  that  speed.  Maybe  this  is  a  story  of 
acquisition  reform  after  all— how  a  single  program  decided  to  reform  itself. 


Schlacter  joined  the  Missile  Defense  Agency  in  2006  and  currently  is  the  deputy  director 
of  the  space  systems  program  office.  He  has  been  the  Spacebased  Kill  Assessment  program 
manager  since  2014. 


39 


Defense  AT&L:  March-April  2018 


Spacebased  Kill  Assessment 

Since  the  1980s,  experts  have  known  the  benefit  of  having  a 
battle  damage  sensor  observe  a  national  missile  defense  en¬ 
gagement.  That  benefit  is  the  ability  to  answer  the  question: 
Did  we  destroy  the  lethal  object? 

To  study  that  question,  the  Ballistic  Missile  Defense  Orga¬ 
nization  conducted  experiments  to  study  the  signature  of  a 
high-speed  collision  in  space.  The  Missile  Defense  Agency 
(MDA)  continued  that  work  by  building  models  that  explained 
the  physics  of  a  high-speed  collision  and  developing  a  test 
sensor  called  the  Kill  Assessment  Sensor  Package,  or  KASP, 
that  supported  maritime  tests  for  over  a  decade. 

In  2014,  MDA  started  Spacebased  Kill  Assessment,  or  SKA, 
with  an  aim  to  provide  a  kill  assessment  capability  from  space. 
At  its  core,  SKA  has  two  fundamental  tenets  in  its  approach: 
First,  it  builds  upon  the  previous  physics-based  collision  mod¬ 
eling  and  the  KASP  sensor  design  heritage  to  reduce  risk.  Sec¬ 
ond,  the  network  of  SKA  sensors  are  to  be  hosted  on  com¬ 
mercial  satellites  as  MDA's  pathfinder  for  an  alternative  to  the 
traditional,  large  satellite  space  program. 

During  the  SKA  design  concept  phase,  it  became  clear  that 
nearly  every  technical  challenge  would  pale  in  comparison  to 
the  schedule  challenge  MDA  would  accept  when  it  selected 
the  host  for  the  SKA  sensors.  Compared  to  a  traditional  DoD 
launch  schedule,  the  commercial  host's  launch  schedule  is 
fixed,  meaning  SKA  sensors  delivered  late  would  miss  their 
ride  into  space. 

Just  as  important,  the  schedule  was  highly  compressed.  For 
example,  it  forced  the  sensor  development  to  be  accomplished 
in  15  months  and  required  MDA  to  deliver  flight  hardware  to 
the  satellite  integrator  starting  at  month  19.  Ultimately,  MDA 
triumphed  over  those  challenges.  This  raises  two  questions: 
(1)  How  did  SKA  become  a  fast  program;  (2)  and  what  were 
the  benefits  of  being  a  fast  program? 

How  To  Be  Fast 

Programs  just  starting  are  in  the  unique  position  of  being  able 
to  establish  a  business  culture  more  effectively  than  programs 
already  under  way.  This  is  because  the  programs  under  way 
cannot  just  establish  a  business  culture— instead  they  have 
the  harder  task  of  changing  one  already  in  use.  The  MDA  lead¬ 
ership  team  exploited  the  opportunity  of  starting  a  program 
with  a  clean  slate  when  it  crafted  the  SKA  business  rules  from 
scratch  in  2014  and  specifically  employed  several  techniques 
to  proceed  rapidly.  The  techniques  listed  below  were  among 
the  most  effective  and  are  portable  to  nonspace  programs. 

Technique  1:  Establish  program  priorities.  Establishing  priori¬ 
ties  seems  like  a  common-sense  approach.  However,  it  came 
as  a  surprise  to  learn  how  few  programs  actually  do  that. 
Without  established  priorities,  program  execution  can  quickly 
become  focused  on  delivery  of  all  the  desired  program  perfor¬ 
mance  with  little  regard  to  cost  or  schedule;  in  other  words,  an 


all-access  pass  to  a  Nunn-McCurdy  breach.  To  guard  against 
that,  on  the  first  day  of  the  program,  the  leadership  team  es¬ 
tablished  the  following  priorities: 

■  Schedule 

■  Cost  containment 

■  Performance 

Schedule  obviously  was  the  highest  priority  because  SKA 
sensors  delivered  late  would  miss  their  ride  into  space;  the 
other  two  priorities  were  assigned  their  order  because  of  their 
alignment  with  the  SKA  acquisition  strategy  and  existing  MDA 
acquisition  culture,  respectively. 

Establishing  priorities  had  the  benefit  of  creating  decision¬ 
making  space  for  the  program  to  trade  cost  or  performance 
to  ease  schedule  pressures.  Having  trade  space  is  absolutely 
paramount  to  program  managers  because  the  best  shape  of 
any  program  will  be  on  the  eve  before  the  program  starts.  As 
soon  as  the  program  is  under  way,  it  cannot  avoid  the  fate 
predicted  by  Helmuth  von  Moltke  the  Elder  that  "no  plan  sur¬ 
vives  contact  with  the  enemy"  and  program  managers  need 
internal  trade  space  to  find  resources  to  apply  to  challenges. 

On  several  occasions,  the  program  spent  funds  (priority  No. 
2)  and  adjusted  performance  (priority  No.  3)  to  stay  inside 
the  schedule  box  (The  No.  1  priority).  In  retrospect,  it  would 
have  been  nearly  impossible  to  go  as  fast  as  the  program  did 
without  having  a  priority-based  system  in  place  to  support 
trade  studies. 

Additionally,  establishing  priorities  gave  the  program  complete 
alignment— among  the  staff  within  the  MDA  team,  among 
the  staff  within  the  development  team  and  between  the  two 
teams.  Team  misalignment  creates  friction  in  the  gears— de¬ 
cision-making  becomes  difficult  and  slows  the  progress  once 
a  decision  is  made.  MDA  observed  that  the  government  and 
developer  teams  worked  faster  by  themselves  and  with  each 
other  once  they  embraced  the  prioritization  schema.  This  ac¬ 
celerated  pace  occurred  because  there  was  never  a  lengthy 
discussion  on  what  the  priority  was  (schedule)  and  what  re¬ 
sources  were  candidates  for  donation  (cost  and  performance). 

All  in  all,  MDA  saw  that  once  the  program  became  more 
comfortable  with  the  priority-based  trade  process,  lengthy 
discussions  and  program  misalignments  occurred  less  and 
less  frequently.  Challenges  were  overcome  at  the  lowest  pos¬ 
sible  level,  and  it  seemed  as  if  the  seas  of  program  execution 
were  becoming  calmer  as  the  program  continued  its  voyage. 

Technique  2:  Unrelenting  pursuit  of  decision-making  speed. 

Traditional  decision-making  routines  can  be  good  because 
they  bring  regularity  to  management  teams.  However,  some 
decision-making  routines  can  be  bad  if  their  cadence  is  too 
slow  and  teams  have  to  wait  too  long  before  the  next  session 
to  make  a  decision,  or  if  the  routines  themselves  generate  un¬ 
necessary  work.  To  that  end,  the  MDA  team  threw  tradition 
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out  the  window  and  put  business  rules  in  place  to  increase 
decision  making  speed,  for  example: 

The  MDA  developer  team  increased  the  frequency  of  deci¬ 
sion-making  forums.  In  lieu  of  monthly  or  quarterly  program 
management  reviews,  the  full  SKA  team  met  weekly  to  make 
decisions.  Each  meeting  was  a  raw,  unvarnished  review  of 
progress,  and  meeting  weekly  minimized  the  amount  of  time 
issues  sat  idle. 


The  government  leadership  team  met  twice  each  week  to  dis¬ 
cuss  "What  did  you  learn  recently?"  This  practice  is  similar  to 
one  commonly  used  in  operational  units  called  "stand  ups." 
This  session  kept  issue  status  up  to  date  and  ensured  that  the 
management  team  shared  a  common  operating  picture. 

SKA  avoided  practices  that  robbed  speed  and  employed  other 
practices  that  increased  speed.  With  government  purpose 
rights  in  place  and  easy-to-use  online  documentation,  shar¬ 
ing  and  storage,  the  program  limited  the  number  of  contractual 
plans  and  reports  to  five.  In  addition,  no  meetings  required 
read-ahead  material  nor  did  any  meetings  require  a  "pre-meet¬ 
ing"  to  get  organized.  Additionally,  the  government  program 
manager  was  onsite  at  the  developer  location  1  day  per  week, 
with  some  engineers  being  onsite  up  to  3  days  per  week,  effec¬ 
tively  creating  a  badgeless  work  environment  where  decisions 
could  be  made  face-to-face  and  nearly  as  quickly  as  sending 
an  e-mail  or  text  message. 

The  Benefits  of  Being  Fast 

Keeping  on  schedule  while  rapidly  delivering  a  capability 
should  be  sufficient  satisfaction.  However,  the  program  also 
noticed  other  benefits  of  being  fast— some  quantifiable,  oth¬ 
ers  intangible. 

Cost  containment.  Early  in  the  program,  a  cost  estimator  told 
the  SKA  leadership  team  that  the  greatest  cost  driver  on  de¬ 
velopment  programs  is  the  salary  of  the  "standing  army"  that 
must  be  paid  even  if  the  schedule  slips  to  the  right.  It  follows 
that  if  a  program  can  stay  on  schedule,  then  cost  containment 
is  easier  to  achieve. 


That  was  true  in  SKA's  experience.  By  staying  on  schedule, 
additional  costs  for  salary  were  contained.  In  fact,  the  overall 
program  cost  increased  less  than  5  percent  with  the  major 
contributor  being  price  increases  for  electronic  parts.  The 
bottom  line  is  that  the  labor  costs  for  the  development  team 
typically  will  surpass  the  costs  for  parts  and  raw  materials, 
fees  for  outsourced  services,  etc.,  so  that  priority  attention  to 
schedule  and  mindfulness  of  the  costs  of  the  standing  army 
will  contribute  greatly  to  cost  control. 

Increased  credibility.  Knowing  how  to  go  fast  allows  pro¬ 
grams  to  create  schedule  reserve  against  unforeseen  chal¬ 
lenges,  and  judiciously  using  that  reserve  allows  programs 
to  stay  on  schedule. 

Staying  on  schedule  can  be  one  of  the  best  ways  to  gain  cred¬ 
ibility  with  those  who  have  a  role  in  DoD  acquisition  pro¬ 
grams— the  joint  warfighters,  Congress,  the  GAO,  etc.  That 
credibility  can  forge  solid  relationships  between  stakeholders 
when  things  are  going  well,  and  can  buy  the  program  additional 
time  when  things  are  not  going  so  well. 

Benefits  to  joint  and  enterprise  partners.  In  today  s  world 
of  increased  interoperability  within  a  joint  or  enterprise  archi¬ 
tecture,  programs  easily  can  be  affected  by  the  performance 
of  their  interfacing  neighbors.  That  means  a  lagging  program 
providing  products  or  services  to  the  enterprise  will  affect  all 
interfacing  programs'  ability  to  meet  schedule  and  therefore 
affect  their  cost  bottom  lines  due  to  the  expense  of  their  own 
standing  armies. 

In  SKA's  experience,  the  speed  the  program  created  had  a 
cascading  effect:  Speed  allowed  SKA  to  keep  schedule,  which 
provided  products  and  services  to  interfacing  programs  on 
the  timeline  it  promised,  therefore  minimizing  cost  impacts 
to  those  same  programs.  In  a  broader  sense  beyond  SKA,  for 
an  agency  managing  a  diverse  enterprise,  the  cost  savings  are 
magnified  and  can  be  impressive  if  all  interfacing  programs  are 
managed  to  this  common  goal  (i.e.,  schedule  speed). 

Conclusion 

SKA  learned  that  immediately  establishing  and  sticking  to  pro¬ 
gram  priorities  as  well  as  an  insistent  pursuit  of  decision-mak¬ 
ing  speed  provide  incredible  program  advantages  that  went 
beyond  the  obvious  advantage  of  maintaining  an  aggressive 
program  schedule. 

Rapid  capability  organizations  are  not  new  in  the  DoD  and 
there  are  numerous  examples  of  how  military  space  programs 
succeeded  in  going  fast.  Their  stories  should  be  read  and  scru¬ 
tinized  for  what  worked  and  what  didn't  work.  The  Spacebased 
Kill  Assessment  story  is  proof  that  nearly  any  program  can  go 
fast— all  it  takes  is  courage,  decisiveness  and  the  willingness  to 
try  something  new  or  different.  Maybe  this  is  what  real  world 
acquisition  reform  looks  like  after  all.  & 

The  author  can  be  contacted  at  Michael. Schlacter@mda. mil. 
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THE  DEFENSE  ACQUISITION  WORKFORCE  AWARDS 


The  Honorable  Ellen  Lord,  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
(USD[AT8cL])  hosted  the  ceremony.  She  emphasized  that  “Defense  acquisition  success  is  critical 
to  providing  the  warfighters  what  they  need  quickly  and  at  the  lowest  possible  cost.  Our  success  is 
dependent  on  the  outstanding  capability  and  performance  of  our  people.”  The  Honorable  Patrick 
Shanahan,  Deputy  Secretary  of  Defense  and  ceremony  distinguished  guest,  echoed  those  senti¬ 
ments,  stating  that  the  acquisition  workforce  will  “help  [the  Department  of  Defense]  become  even 
more  competitive  than  we  are  today.” 

The  Defense  Acquisition  Workforce  Individual  Achievement  Awards  highlight  premier  individuals 
who  demonstrate  the  highest  levels  of  excellence  and  professionalism  in  the  acquisition  of  prod¬ 
ucts  and  services  for  the  Department  of  Defense.  These  awards  recognize  individuals  in  each  of  the 
acquisition  functional  disciplines. 

The  Defense  Acquisition  Workforce  Development  Innovation  Awards  were  established  to  recog¬ 
nize  excellence  by  acquisition  organizations  in  developing  unique  and  innovative  solutions  to  en¬ 
sure  that  their  workforce  is  well  equipped  to  deliver  world-class  capabilities  to  the  warfighter. 
The  award  highlights  demonstrated  exceptional  outside-the-box  thinking  and  progress  in  tackling 
workforce  development  challenges. 
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Photos  by  Specialist  Tammy  Nooner,  USA. 


Mr.  Lee  R.  Rosenberg 
Missile  Defense  Agency  | 
Small  Business 


INDIVIDUAL  ACHIEVEMENT  AWARD  WINNERS 

The  Honorable  Ellen  Lord  presented  the  awards,  accompanied  by  the  Honorable  Patrick  Shanahan;  and 
Lieutenant  General  Anthony  Ierardi,  U.S.  Army,  Director,  Force  Structure,  Resources  and  Assessment  (J8) 
Joint  Staff. 


Lieutenant  Colonel 

Thomas  A.  Atkinson 

U.S.  Special  Operations  Command  | 

Program  Management 


Mr.  Ulises  Cartaya 
U.S.  Special  Operations  Command  | 
Services  Acquisition 
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Mr.  Thomas  Sachse 
U.S.  Navy  |  Test  and  Evaluation 


Ms.  Melissa  Panarelli 

Defense  Contract  Audit  Agency 

Auditing 


Mr.  Skip  Hinman 

U.S.  Air  Force  |  Requirements 

Management 


Gunnery  Sergeant 
Tamalia  C.  Adams 
U.S.  Marine  Corps  | 
Acquisition  in  an 
Expeditionary  Environment 
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Mr.  Gary  V.  Trimble 
U.S.  Navy  |  Contracting  and 
Procurement 


Dr.  Wilson  Rosa 
U.S.  Navy  | 

Cost  Estimating 


Lieutenant  Colonel 
Andrew  R.  Vrabec 
U.S.  Air  Force  |  Earned 
Value  Management 
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Mr.  Daniel  M.  Carroll 
U.S.  Special  Operations 
Command  |  Engineering 


Ms.  Valerie  Clinkenbeard 
U.S.  Army  | 
Facilities  Engineering 


Mr.  Jeffrey  M.  Martin 
U.S.  Air  Force  |  Financial 
Management 


Defense  AT&L:  March-April  2018 


Mr.  George  N.  Graham,  Jr. 
U.S.  Special  Operations  Command  | 
Life  Cycle  Logistics 


Dr.  Charles  A.  Bass,  Jr. 
U.S.  Army  |  Science  and 
Technology  Manager 


Captain  Charles  M.  Stuart 

U.S.  Navy  |  Production,  Quality,  and 

Manufacturing 


Mr.  Michael  R.  Cirillo 
U.S.  Marine  Corps  | 
Information  Technology 


47 


Defense  AT&L:  March-April  2018 


-rtlarcte  (9m, anization 


’ mall  (9manizatio, 


DEVELOPMENT  INNOVATION  AWARD  WINNERS 


Defense  Contract  Management  Agency,  Fort  Lee,  Virginia 

The  Honorable  Ellen  Lord;  Kathleen  Butera;  Marie  Greening;  Chris  Zubof;  the  Honorable  Patrick 
Shanahan,  Deputy  Secretary  of  Defense;  and  Lieutenant  General  Anthony  Ierardi,  Director,  Force 
Structure,  Resources  and  Assessment  ( J8),  Joint  Staff 


U.S.  Army  Contracting  Command,  Orlando,  Florida 

Front  Row:  Patricia  Neal,  Kimberly  Tedeschi,  Servola  Frazier,  Rosa  Rivera,  Jeff  Claar 
Back  Row:  Kristie  Brandon;  the  Honorable  Ellen  Lord;  Joseph  Giunta,  the  Honorable  Patrick 
Shanahan;  and  Lieutenant  General  Anthony  Ierardi. 

Photos  by  Dirke  Williams,  Department  of  Defense. 
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